New submission from Alecsandru Patrascu:
Hi All,
This is Alecsandru from Server Scripting Languages Optimization team at Intel
Corporation.
I would like to submit a request to turn-on Profile Guided Optimization or PGO
as the default build option for Python (both 2.7 and 3.6), given its
Alecsandru Patrascu added the comment:
I received the review and will post new patch versions as soon as I update
them.
Regarding PGO on clang, I will need a bit more time to edit the Makefile and
will post it just for clang, to be easier for us to see the differences
Changes by Alecsandru Patrascu alecsandru.patra...@intel.com:
Removed file: http://bugs.python.org/file40226/python2.7-3.6-pgo.zip
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24915
Changes by Alecsandru Patrascu alecsandru.patra...@intel.com:
--
keywords: +patch
Added file: http://bugs.python.org/file40231/python2.7-pgo.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24915
Changes by Alecsandru Patrascu alecsandru.patra...@intel.com:
Added file: http://bugs.python.org/file40232/python3.6-pgo.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24915
Alecsandru Patrascu added the comment:
The patches are tested on Linux machines, with GNU GCC 4.8.3. From your output
I see that you are using the CLANG compiler. CLANG uses a different set of
flags for PGO that are not compatible with GCC's, therefore the compilation
will fail. Can you
Alecsandru Patrascu added the comment:
I added the patches as individual files and removed the zip file.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24915
Changes by Alecsandru Patrascu alecsandru.patra...@intel.com:
Added file: http://bugs.python.org/file40238/python2.7-pgo-v02-mac.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24915
Alecsandru Patrascu added the comment:
I modified the patches after the review made by Brett (python2.7-pgo-v02.patch
and python3.6-pgo-v02.patch):
- removed the call to pybench
- left the PGO steps as optional. To use it we run make profile-opt
- in the initial patches, I left out test_hashlib
Changes by Alecsandru Patrascu alecsandru.patra...@intel.com:
Added file: http://bugs.python.org/file40237/python3.6-pgo-v02.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24915
Changes by Alecsandru Patrascu alecsandru.patra...@intel.com:
Added file: http://bugs.python.org/file40236/python2.7-pgo-v02.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24915
Changes by Alecsandru Patrascu alecsandru.patra...@intel.com:
Added file: http://bugs.python.org/file40239/python3.6-pgo-v02-mac.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24915
Changes by Alecsandru Patrascu alecsandru.patra...@intel.com:
Added file: http://bugs.python.org/file40255/python3.5-pgo-v03.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24915
Changes by Alecsandru Patrascu alecsandru.patra...@intel.com:
Added file: http://bugs.python.org/file40254/python2.7-pgo-v03.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24915
Alecsandru Patrascu added the comment:
I modified the patches to be compatible with both environments. The new
versions modify the configure.ac file also, therefore you will need to run
autoconf by hand. Also, in case of MaOS you will need to have llvm-profdata
installed and in your path.
I
Changes by Alecsandru Patrascu alecsandru.patra...@intel.com:
Added file: http://bugs.python.org/file40268/python3.6-pgo-v04.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24915
Changes by Alecsandru Patrascu alecsandru.patra...@intel.com:
Added file: http://bugs.python.org/file40267/python2.7-pgo-v04.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24915
Changes by Alecsandru Patrascu alecsandru.patra...@intel.com:
Removed file: http://bugs.python.org/file40263/python3.6-pgo-v04.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24915
Changes by Alecsandru Patrascu alecsandru.patra...@intel.com:
Removed file: http://bugs.python.org/file40262/python2.7-pgo-v04.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24915
Alecsandru Patrascu added the comment:
The profile merging is necessary in case you want to use a pure clang compiler
or you use GCC in OSX. For example, a general profiling action using clang will
result in at least one binary profile. For our case, when using regrtest, we
will have multiple
Changes by Alecsandru Patrascu alecsandru.patra...@intel.com:
Added file: http://bugs.python.org/file40289/python3.6-pgo-v06.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24915
Changes by Alecsandru Patrascu alecsandru.patra...@intel.com:
Added file: http://bugs.python.org/file40288/python2.7-pgo-v06.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24915
Alecsandru Patrascu added the comment:
I've updated the patches after review and implemented the checkup for
llvm-profdata for both Linux and OSX.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24915
Changes by Alecsandru Patrascu alecsandru.patra...@intel.com:
Added file: http://bugs.python.org/file40299/README2.7-pgo-v01.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24915
Changes by Alecsandru Patrascu alecsandru.patra...@intel.com:
Added file: http://bugs.python.org/file40300/README3.6-pgo-v01.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24915
Alecsandru Patrascu added the comment:
That's a good point Skip. I added another set of patches, just for the README
files, explaining the entire procedure, so now anyone reading it will see that
PGO is available, what are the steps involved and a brief comment about the
warning
Changes by Alecsandru Patrascu alecsandru.patra...@intel.com:
Added file: http://bugs.python.org/file40262/python2.7-pgo-v04.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24915
Alecsandru Patrascu added the comment:
I modified the patches with clang support.
Also, I added an important check for the architecture on which PGO is running.
Our proposal targets x86 platforms, since our measurements are made only on x86
hardware
Changes by Alecsandru Patrascu alecsandru.patra...@intel.com:
Added file: http://bugs.python.org/file40263/python3.6-pgo-v04.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24915
Changes by Alecsandru Patrascu alecsandru.patra...@intel.com:
Added file: http://bugs.python.org/file40274/python3.6-pgo-v05.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24915
Changes by Alecsandru Patrascu alecsandru.patra...@intel.com:
Added file: http://bugs.python.org/file40273/python2.7-pgo-v05.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24915
Alecsandru Patrascu added the comment:
I fixed the files after the review. Regarding the PROFILE_OPT_OLD line, I think
that it is better to keep also the old task used for PGO, until clear evidence
and measurements that regrtest is performing better on other architectures
exists
Alecsandru Patrascu added the comment:
If you are talking just about the 3.6 patch, it is called this way to emphasize
the fact that it is intended for the development branch. It is perfectly
compatible with 3.5, therefore it is not needed for packagers to maintain two
distinct versions. I've
Alecsandru Patrascu added the comment:
I've updated the patches with redirect to /dev/null, as is it is more clearer
to the user what is our intent, without having him to necessarily read the
regrtest documentation. I've also added a warning message regarding the output
and ported all
Alecsandru Patrascu added the comment:
I named this task PROFILE_TASK_X86 because it is rigorously tested and we have
proven that regrtest performs better on this architecture. Until any other
clear evidence and solid measurements that regrtest is performing better on
other architectures
Changes by Alecsandru Patrascu <alecsandru.patra...@intel.com>:
Added file: http://bugs.python.org/file40387/python2.7-pgo-v07.patch
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python
Changes by Alecsandru Patrascu <alecsandru.patra...@intel.com>:
Added file: http://bugs.python.org/file40388/python3.6-pgo-v07.patch
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python
Changes by Alecsandru Patrascu <alecsandru.patra...@intel.com>:
Added file: http://bugs.python.org/file40389/README2.7-pgo-v02.patch
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python
Changes by Alecsandru Patrascu <alecsandru.patra...@intel.com>:
Added file: http://bugs.python.org/file40390/README3.6-pgo-v02.patch
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python
Changes by Alecsandru Patrascu <alecsandru.patra...@intel.com>:
Removed file: http://bugs.python.org/file40387/python2.7-pgo-v07.patch
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python
Changes by Alecsandru Patrascu <alecsandru.patra...@intel.com>:
Added file: http://bugs.python.org/file40392/python3.6-pgo-v07.patch
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python
Changes by Alecsandru Patrascu <alecsandru.patra...@intel.com>:
Removed file: http://bugs.python.org/file40388/python3.6-pgo-v07.patch
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python
Changes by Alecsandru Patrascu <alecsandru.patra...@intel.com>:
Added file: http://bugs.python.org/file40391/python2.7-pgo-v07.patch
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python
Changes by Alecsandru Patrascu <alecsandru.patra...@intel.com>:
Added file: http://bugs.python.org/file40460/hash8-v02.patch
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python
Alecsandru Patrascu added the comment:
I've submitted a more compact version of the previous code (hash8-v02.patch)
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Alecsandru Patrascu added the comment:
Yes, it doesn't break the compatibility with existing applications. To make
sure, we tested it in various scenarios and workloads.
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.p
Alecsandru Patrascu added the comment:
Please hold this patch for a while, while I analyze the received feedback.
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
New submission from Alecsandru Patrascu:
Hi All,
This is Alecsandru from Server Scripting Languages Optimization team at Intel
Corporation.
I would like to submit a patch that improves the performance of the hash
computation code on stringobject, bufferobject and unicodeobject. As can
Alecsandru Patrascu added the comment:
Thank you for the tweaks. I tested the patch and it is all working fine.
Without failure count the PGO run is even more transparent to the user.
I tested your final version of the patch on several Linux server and regular
desktop distributions
Alecsandru Patrascu added the comment:
Thank you Brett for your effort to merge all the changes made to regrtest
despite the changes appeared on the way!
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Alecsandru Patrascu added the comment:
Thank you for upstreaming this in both branches of Python!
Do you think that a different version of regrtest.py, that will be used only
for PGO training, should be better in this case? I mean, by implementing a
custom version, I think we can control
New submission from Alecsandru Patrascu:
This issue adds improved support for Profile Guided Optimization builds in the
regression test suite.
Mainly, we keep the visual progress status in this process and suppress errors
or any other unnecessary output that can alarm users
Alecsandru Patrascu added the comment:
I can work on modifying the existing regrtest and adding a distinct flag, --pgo
for example, as Antoine suggested. Indeed, it will not be trivial as regrtest
has a dual approach (single process and multi process), but I will give it a
try and post
Changes by Alecsandru Patrascu <alecsandru.patra...@intel.com>:
Added file: http://bugs.python.org/file40547/regrtest_36_v02.patch
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python
Alecsandru Patrascu added the comment:
The patch does not display dots, as they are not that intuitive and users
cannot make an estimate of the time left or the position in the training phase.
Instead, the regrtest.py, will display the progress as [X/Y], like the default
version, but without
Changes by Alecsandru Patrascu <alecsandru.patra...@intel.com>:
--
keywords: +patch
Added file: http://bugs.python.org/file40532/regrtest_27.patch
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python
Changes by Alecsandru Patrascu <alecsandru.patra...@intel.com>:
Added file: http://bugs.python.org/file40533/regrtest_36.patch
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python
Alecsandru Patrascu added the comment:
I've added the patches for Python default and 2.7. To apply them please use the
command "hg import --no-commit". PGO flag is implemented for both variants
(single and multi process). Default in profile-opt will be sing
Alecsandru Patrascu added the comment:
Hi, I added a dedicated issue just for LTO only when using GCC and CLANG
(http://bugs.python.org/issue25702), that works well with PGO also.
--
nosy: +alecsandru.patrascu
___
Python tracker <
Alecsandru Patrascu added the comment:
Thank you David for including me in this issue.
On ICC, when executing the instrumented applications, it will generate dynamic
information file that has a unique name and .dyn suffix. From my previous
experience with GCC and CLANG PGO patch (issue24915
Alecsandru Patrascu added the comment:
Hello, I've added an updated set of patches (v03) for the current CPython2 and
CPython3 codebase. Also made some small changes to reduce the number of places
where the flags are set.
--
Added file: http://bugs.python.org/file41493/lto-cpython2
Changes by Alecsandru Patrascu <alecsandru.patra...@intel.com>:
Added file: http://bugs.python.org/file41494/lto-cpython3-v03.patch
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python
Alecsandru Patrascu added the comment:
Thank you for the reviews. I've added a new patch (v02) to correct the issues.
--
Added file: http://bugs.python.org/file41386/coverage-2.7-v02.patch
___
Python tracker <rep...@bugs.python.org>
Alecsandru Patrascu added the comment:
I'm adding Brett, Gregory, Stefan and Victor as nosy because this issue might
be interesting for them also.
--
nosy: +brett.cannon, gregory.p.smith, haypo, scoder, skrah
___
Python tracker <
New submission from Alecsandru Patrascu:
Hi All,
This is Alecsandru from Server Scripting Languages Optimization team at Intel
Corporation. I would like to submit a patch that adds extended support for
coverage, in the same way that it is used in CPython3. Used in conjunction
Alecsandru Patrascu added the comment:
LTO exists in GCC since version 4.5, but it is true that only recent versions
(>=4.8) perform it in good conditions. It is not enabled by default in this
patch, it is only available when building with PGO support. Running just "make"
will
New submission from Alecsandru Patrascu:
Title: Link Time Optimizations support for GCC and CLANG
Hi All,
This is Alecsandru from Server Scripting Languages Optimization team at Intel
Corporation. I would like to submit a patch that adds support for Link Time
Optimization (LTO) when using
Changes by Alecsandru Patrascu <alecsandru.patra...@intel.com>:
Added file: http://bugs.python.org/file41131/lto-cpython3-v01.patch
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python
Changes by Alecsandru Patrascu <alecsandru.patra...@intel.com>:
--
keywords: +patch
Added file: http://bugs.python.org/file41130/lto-cpython2-v01.patch
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python
Changes by Alecsandru Patrascu <alecsandru.patra...@intel.com>:
Added file: http://bugs.python.org/file41138/lto-cpython3-v02.patch
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python
Alecsandru Patrascu added the comment:
Meanwhile I've added the patches (v02) for LTO enabled only if the "./configure
--with-lto" command is issued.
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.py
Changes by Alecsandru Patrascu <alecsandru.patra...@intel.com>:
Added file: http://bugs.python.org/file41137/lto-cpython2-v02.patch
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python
Alecsandru Patrascu added the comment:
I've updated the patch for 2.7 branch to contain the LTO part merged by Gregory.
--
Added file:
http://bugs.python.org/file43142/cpython2_with_optimizations_v02.patch
___
Python tracker <rep...@bugs.python.
Alecsandru Patrascu added the comment:
The patches in discussions are the v02 ones.
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Alecsandru Patrascu added the comment:
Hello Brett, I saw that there was no other activity on this issue so far. Can
you please update this, as I do not have rights?
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Alecsandru Patrascu added the comment:
Sure, I attached them as files because they have a lot of lines for posting
here (~90 in total).
The linker offers the possibility to show what piece of data/functions was
removed, but I intentionally omitted it in order not to clutter the build
trace
Changes by Alecsandru Patrascu <alecsandru.patra...@intel.com>:
Added file: http://bugs.python.org/file41809/gc-removed-cpython3.txt
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python
Alecsandru Patrascu added the comment:
Hello and thank you for your feedback. For CPython this does not apply because
due to the structure of the build system, inside the "build" directory there
are no PGO profiles saved. You can run find . -name '*.gc
Alecsandru Patrascu added the comment:
Thank you for noticing and signaling this issue. Since I proposed the PGO
patches, I will fix it in another patch, as I don't want it to break any builds.
--
___
Python tracker <rep...@bugs.python.org>
Changes by Alecsandru Patrascu <alecsandru.patra...@intel.com>:
--
nosy: +alecsandru.patrascu
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python
Alecsandru Patrascu added the comment:
That's interesting. Even on CPython3, I still don't see any gcda's inside the
build directory, nor the tree structure you are seeing there. Can you please
give me a couple of details regarding your environment (os, distribution, gcc
version, 32/64 bit
Alecsandru Patrascu added the comment:
For my responses, I modified locally the Makefile so that it will not remove
the build directory and any of the gcda files. I will make some more tests
tomorrow, but i think that this problem will solve simpler if the removal of
the build directory
Changes by Alecsandru Patrascu <alecsandru.patra...@intel.com>:
Added file: http://bugs.python.org/file41806/cpython3-deadcode-v01.patch
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python
New submission from Alecsandru Patrascu:
Hi all,
This is Alecsandru from the Dynamic Scripting Languages Optimization Team at
Intel Corporation. I would like to submit a patch that enables garbage
collection of unused input sections from the CPython2 and CPython3 binaries, by
using the "
Alecsandru Patrascu added the comment:
I've done again the experiments on larger workloads, such as our OpenStack
Swift cluster, and it works without any issues.
Also, I've attached an archive with a simple external module in CPython3 that
uses PyMem_RawMalloc. The output is ok, and it's
Alecsandru Patrascu added the comment:
I realize now that I should have explained a bit more the background of this
patch. I'll do this now, for everyone to be clear of what is the effect of
those flags.
This issue was revealed after running the coverage target over various
workloads
Changes by Alecsandru Patrascu <alecsandru.patra...@intel.com>:
--
nosy: +alecsandru.patrascu
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python
Changes by Alecsandru Patrascu <alecsandru.patra...@intel.com>:
--
nosy: +gregory.p.smith, lemburg, scoder, steve.dower, zach.ware
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python
Changes by Alecsandru Patrascu <alecsandru.patra...@intel.com>:
--
nosy: +brett.cannon, gregory.p.smith, lemburg, pitrou, r.david.murray, scoder,
skrah, steve.dower, zach.ware
___
Python tracker <rep...@bugs.python.org>
<http://
Changes by Alecsandru Patrascu <alecsandru.patra...@intel.com>:
Added file: http://bugs.python.org/file41666/lto-cpython3-v04.patch
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python
Alecsandru Patrascu added the comment:
Thank you for your feedback, I've updated the patches and now LTO flags are
used only when building with PGO (v04). CFLAGS/LDFLAGS remain untouched, as
Antoine and Victor suggested is better.
--
Added file: http://bugs.python.org/file41665/lto
Alecsandru Patrascu added the comment:
As Steve mentioned, the Microsoft compiler uses LTO (they call it Link-Time
Code Generation) and the flags are used when compiling CPython on Windows
systems. Thus our proposal to enable it on GCC and CLANG also
Changes by Alecsandru Patrascu <alecsandru.patra...@intel.com>:
--
nosy: +alecsandru.patrascu
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python
Changes by Alecsandru Patrascu <alecsandru.patra...@intel.com>:
--
nosy: +alecsandru.patrascu
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python
Changes by Alecsandru Patrascu <alecsandru.patra...@intel.com>:
--
resolution: fixed ->
status: closed -> open
___
Python tracker <rep...@bugs.python.org>
<http://bugs.
Alecsandru Patrascu added the comment:
I've added a fix for the PGO builds after the issue pointed out by you. Can you
please test it also?
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
New submission from Alecsandru Patrascu:
Hello,
This is Alecsandru from the Dynamic Scripting Languages Optimization Team at
Intel Corporation. I would like to submit a patch as a response to the
discussion thread opened in Python-Dev
(https://mail.python.org/pipermail/python-dev/2016
Changes by Alecsandru Patrascu <alecsandru.patra...@intel.com>:
Added file: http://bugs.python.org/file41919/cpython3_pgo_default-v02.patch
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python
Changes by Alecsandru Patrascu <alecsandru.patra...@intel.com>:
Added file: http://bugs.python.org/file41917/pgofix-cpython3.patch
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python
Alecsandru Patrascu added the comment:
I've added a fix for the PGO builds after a issue pointed out in #26307. Thank
you Christos for your observation!
--
Added file: http://bugs.python.org/file41916/pgofix-cpython2.patch
___
Python tracker <
1 - 100 of 123 matches
Mail list logo
