[issue25702] Link Time Optimizations support for GCC and CLANG

Change by Inada Naoki : -- stage: patch review -> resolved status: open -> closed ___ Python tracker ___ ___ Python-bugs-list

[issue25702] Link Time Optimizations support for GCC and CLANG

Change by Brett Cannon : -- nosy: -brett.cannon ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue25702] Link Time Optimizations support for GCC and CLANG

Change by David Filiatrault : -- nosy: +David Filiatrault ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue25702] Link Time Optimizations support for GCC and CLANG

Gregory P. Smith added the comment: the main part of this issue is done but it can't be closed until the dependencies listed are also dealt with. un-assigning myself. -- assignee: gregory.p.smith -> resolution: -> fixed ___ Python tracker

[issue25702] Link Time Optimizations support for GCC and CLANG

Roundup Robot added the comment: New changeset f710dac07312 by Gregory P. Smith in branch '2.7': Issue #25702: A --with-lto configure option has been added that will https://hg.python.org/cpython/rev/f710dac07312 -- ___ Python tracker

[issue25702] Link Time Optimizations support for GCC and CLANG

Changes by Gregory P. Smith : -- dependencies: +test_gdb fails all tests on a profile-opt build configured --with-lto ___ Python tracker

[issue25702] Link Time Optimizations support for GCC and CLANG

Changes by Gregory P. Smith : -- dependencies: +test_distutils fails when configured --with-lto ___ Python tracker ___

[issue25702] Link Time Optimizations support for GCC and CLANG

Gregory P. Smith added the comment: What i committed for 3.5 and 3.6 matches lto-cpython3-v04.patch which just adds --with-lto support. 2.7 still needs to be patched. For reference: Using ubuntu's gcc 5.2.1 i was seeing a 2-3% performance increase in the resulting LTO binary vs a plain

[issue25702] Link Time Optimizations support for GCC and CLANG

Roundup Robot added the comment: New changeset f16ec63055ad by Gregory P. Smith in branch '3.5': Issue #25702: A --with-lto configure option has been added that will https://hg.python.org/cpython/rev/f16ec63055ad New changeset 3103af76f4c4 by Gregory P. Smith in branch 'default': Issue #25702:

[issue25702] Link Time Optimizations support for GCC and CLANG

Alecsandru Patrascu added the comment: Maybe an workflow like the one proposed in issue #26359 can be helpful in these development phases. -- ___ Python tracker

[issue25702] Link Time Optimizations support for GCC and CLANG

Stefan Krah added the comment: On Thu, Apr 14, 2016 at 08:55:25AM +, Stefan Krah wrote: > I use it all the time in development: ... where "it" refers to "./configure && make", not to PGO. -- ___ Python tracker

[issue25702] Link Time Optimizations support for GCC and CLANG

Marc-Andre Lemburg added the comment: On 14.04.2016 10:39, Alecsandru Patrascu wrote: > > @Stefan and @Marc, you say that people don't want to wait for PGO to build > when running ./configure && make, but why? Even though many developers use > it, this mode is not intended for development, it

[issue25702] Link Time Optimizations support for GCC and CLANG

Stefan Krah added the comment: On Thu, Apr 14, 2016 at 08:39:20AM +, Alecsandru Patrascu wrote: > @Stefan and @Marc, you say that people don't want to wait for PGO to build > when running ./configure && make, but why? Even though many developers use > it, this mode is not intended for

[issue25702] Link Time Optimizations support for GCC and CLANG

Alecsandru Patrascu added the comment: @Stefan and @Marc, you say that people don't want to wait for PGO to build when running ./configure && make, but why? Even though many developers use it, this mode is not intended for development, it is production level and should be run once (or at leas

[issue25702] Link Time Optimizations support for GCC and CLANG

Antoine Pitrou added the comment: Le 13/04/2016 21:55, Marc-Andre Lemburg a écrit : >> >> LTO is not stable on all platforms (according to doko), and people don't >> want to wait for PGO to build when they just run ./configure && make. >> >> --with-pgo and --with-lto is fine. > > Agreed. Let's

[issue25702] Link Time Optimizations support for GCC and CLANG

Marc-Andre Lemburg added the comment: On 13.04.2016 21:50, Stefan Krah wrote: > > LTO is not stable on all platforms (according to doko), and people don't > want to wait for PGO to build when they just run ./configure && make. > > --with-pgo and --with-lto is fine. Agreed. Let's not make

[issue25702] Link Time Optimizations support for GCC and CLANG

STINNER Victor added the comment: > LTO is not stable on all platforms (according to doko), and people don't want > to wait for PGO to build when they just run ./configure && make. Can we have a whitelist of arch known to support PGO and/or LTO? Or maybe a blacklist? Ubuntu already has this

[issue25702] Link Time Optimizations support for GCC and CLANG

Stefan Krah added the comment: LTO is not stable on all platforms (according to doko), and people don't want to wait for PGO to build when they just run ./configure && make. --with-pgo and --with-lto is fine. -- ___ Python tracker

[issue25702] Link Time Optimizations support for GCC and CLANG

STINNER Victor added the comment: + --with-lto Enable Link Time Optimization in PGO builds. + Disabled by default. I don't understand why it's disabled by default. IMHO we must enable all the best optimizers options *by default*. But I expect all

[issue25702] Link Time Optimizations support for GCC and CLANG

Gregory P. Smith added the comment: Piping up from the peanut gallery here: If your use case is not doing release builds for production use, i.e. "casual use", don't bother with either PGO or LTO. It won't matter. Your final build that you Q ship should absolutely use those. (nobody's going

[issue25702] Link Time Optimizations support for GCC and CLANG

INADA Naoki added the comment: > For sure you are not the only user that has active workloads on the physical > machine while you do benchmarks :) I think largest machine type I chosen (32core) can avoid sharing physical machine with other users. > On the other hand, the path you are going

[issue25702] Link Time Optimizations support for GCC and CLANG

Alecsandru Patrascu added the comment: You are doing measurements on a virtual machine... For sure you are not the only user that has active workloads on the physical machine while you do benchmarks :) On the other hand, the path you are going with just LTO is nice for experiments, but for

[issue25702] Link Time Optimizations support for GCC and CLANG

INADA Naoki added the comment: The machine is Google Compute Engine n1-highcpu-32 (Intel Ivy Bridge) Linux bench 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3 (2016-01-17) x86_64 GNU/Linux cpuinfo: processor : 31 vendor_id : GenuineIntel cpu family : 6 model :

[issue25702] Link Time Optimizations support for GCC and CLANG

Alecsandru Patrascu added the comment: >From our experience, pybench only is not a representative benchmark. Instead, >if you like to measure performance close to real workloads, you can run the >Grand Unified Python Benchmark suite, that is more complete. Also, you need to take into

[issue25702] Link Time Optimizations support for GCC and CLANG

INADA Naoki added the comment: I've tried LTO without PGO in Debian Jessie. $ LTOFLAGS='-flto -fuse-linker-plugin -ffat-lto-objects -flto-partition=none' $ CFLAGS=$LTOFLAGS LDFLAGS=$LTOFLAGS ./configure --prefix=... $ make -j32 results is here (compared with neither LTO and PGO): Test

[issue25702] Link Time Optimizations support for GCC and CLANG

Alecsandru Patrascu added the comment: I understand now your question. LTO is not enabled when running just `make`, only in `make profile-opt` -- ___ Python tracker

[issue25702] Link Time Optimizations support for GCC and CLANG

INADA Naoki added the comment: Sorry my poor English. I meant that "Does `./configure --with-lto && make` use LTO?". -- ___ Python tracker ___

[issue25702] Link Time Optimizations support for GCC and CLANG

Alecsandru Patrascu added the comment: Yes, you can use LTO without PGO, but the proposed ways it's more efficient and makes more sense for CPython builds. -- ___ Python tracker

[issue25702] Link Time Optimizations support for GCC and CLANG

Alecsandru Patrascu added the comment: As Steve mentioned, the Microsoft compiler uses LTO (they call it Link-Time Code Generation) and the flags are used when compiling CPython on Windows systems. Thus our proposal to enable it on GCC and CLANG also. --

[issue25702] Link Time Optimizations support for GCC and CLANG

Steve Dower added the comment: MSVC has had Link-Time Code Generation for many releases, and it should have been used for all 2.7 releases (definitely used in 3.5+) to optimize references between object files. I assume this is equivalent to LTO. We currently don't use PGO in the official

[issue25702] Link Time Optimizations support for GCC and CLANG

Brett Cannon added the comment: To help answer MAL's question about Windows: I know the python.org installers are **not** built with PGO, but I don't know about LTO. -- nosy: +steve.dower ___ Python tracker

[issue25702] Link Time Optimizations support for GCC and CLANG

STINNER Victor added the comment: I suggest to only modify the default branch and work with downstream (like Linux vendors) to compile Python with best compiler options. I'm talking about the default compilation mode. Maybe we can add a configure option to 2.7 and 3.5, disabled by default, to

[issue25702] Link Time Optimizations support for GCC and CLANG

Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file41666/lto-cpython3-v04.patch ___ Python tracker ___

[issue25702] Link Time Optimizations support for GCC and CLANG

Alecsandru Patrascu added the comment: Thank you for your feedback, I've updated the patches and now LTO flags are used only when building with PGO (v04). CFLAGS/LDFLAGS remain untouched, as Antoine and Victor suggested is better. -- Added file:

[issue25702] Link Time Optimizations support for GCC and CLANG

Marc-Andre Lemburg added the comment: On 20.01.2016 09:37, Antoine Pitrou wrote: > Let me add that downstream distributors already customize compilation options > (Ubuntu's Python is compiled with both PGO and LTO enabled, AFAIR), so this > change may only really affect the tiny subset of

[issue25702] Link Time Optimizations support for GCC and CLANG

Antoine Pitrou added the comment: > If not, then the patch would also effect the not-so-tiny fraction of Python users on Windows ;-) I don't see how enabling LTO for gcc and clang could ever affect our Windows users ;-) -- ___ Python tracker

[issue25702] Link Time Optimizations support for GCC and CLANG

Marc-Andre Lemburg added the comment: On 20.01.2016 11:38, Antoine Pitrou wrote: >> If not, then the patch would also effect the not-so-tiny fraction >> of Python users on Windows ;-) > > I don't see how enabling LTO for gcc and clang could ever affect our Windows > users ;-) You have a point

[issue25702] Link Time Optimizations support for GCC and CLANG

Changes by STINNER Victor : -- nosy: -haypo ___ Python tracker ___ ___

[issue25702] Link Time Optimizations support for GCC and CLANG

R. David Murray added the comment: My understanding is that we (starting with Guido) have made a blanket exception for 2.7 for useful performance and build-system-only related patches. That doesn't mean *anything* can go in (the usual rules about "is this worth it/backward compatible/won't

[issue25702] Link Time Optimizations support for GCC and CLANG

Antoine Pitrou added the comment: > Unfortunately, 2.7 doesn't have even CFLAGS_NODIST; I suspect it may be time > to backport that. I don't think now is a good time to introduce instability in the 2.7 branch. -- ___ Python tracker

[issue25702] Link Time Optimizations support for GCC and CLANG

Zachary Ware added the comment: I'm a bit concerned that the flags are being added unconditionally to CFLAGS and LDFLAGS (when configured --with-lto), which means extensions are forced into it as well. I think it would be better to use CFLAGS_NODIST and to add LDFLAGS_NODIST. Unfortunately,

[issue25702] Link Time Optimizations support for GCC and CLANG

Changes by R. David Murray : -- nosy: +r.david.murray ___ Python tracker ___ ___

[issue25702] Link Time Optimizations support for GCC and CLANG

Zachary Ware added the comment: Unless I'm just missing something, I don't see how introducing CFLAGS_NODIST and LDFLAGS_NODIST to 2.7 would introduce instability. It should be a fairly non-invasive change, restricted to configure and the Makefile; both vars should usually be empty and thus

[issue25702] Link Time Optimizations support for GCC and CLANG

Alecsandru Patrascu added the comment: Hello, I've added an updated set of patches (v03) for the current CPython2 and CPython3 codebase. Also made some small changes to reduce the number of places where the flags are set. -- Added file:

[issue25702] Link Time Optimizations support for GCC and CLANG

Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file41494/lto-cpython3-v03.patch ___ Python tracker ___

[issue25702] Link Time Optimizations support for GCC and CLANG

Alecsandru Patrascu added the comment: I'm adding Brett, Gregory, Stefan and Victor as nosy because this issue might be interesting for them also. -- nosy: +brett.cannon, gregory.p.smith, haypo, scoder, skrah ___ Python tracker

[issue25702] Link Time Optimizations support for GCC and CLANG

Antoine Pitrou added the comment: LTO only exists on recent versions of gcc, so the configure script should probably do some version checking. Also we can't enable it by default as 1) it makes compile times much longer 2) there are some bugs in some gcc versions (see e.g.

[issue25702] Link Time Optimizations support for GCC and CLANG

Alecsandru Patrascu added the comment: LTO exists in GCC since version 4.5, but it is true that only recent versions (>=4.8) perform it in good conditions. It is not enabled by default in this patch, it is only available when building with PGO support. Running just "make" will not activate

[issue25702] Link Time Optimizations support for GCC and CLANG

New submission from Alecsandru Patrascu: Title: Link Time Optimizations support for GCC and CLANG Hi All, This is Alecsandru from Server Scripting Languages Optimization team at Intel Corporation. I would like to submit a patch that adds support for Link Time Optimization (LTO) when using

[issue25702] Link Time Optimizations support for GCC and CLANG

Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file41131/lto-cpython3-v01.patch ___ Python tracker ___

[issue25702] Link Time Optimizations support for GCC and CLANG

Changes by Alecsandru Patrascu : -- keywords: +patch Added file: http://bugs.python.org/file41130/lto-cpython2-v01.patch ___ Python tracker

[issue25702] Link Time Optimizations support for GCC and CLANG

Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file41138/lto-cpython3-v02.patch ___ Python tracker ___

[issue25702] Link Time Optimizations support for GCC and CLANG

Alecsandru Patrascu added the comment: Meanwhile I've added the patches (v02) for LTO enabled only if the "./configure --with-lto" command is issued. -- ___ Python tracker

[issue25702] Link Time Optimizations support for GCC and CLANG

Changes by Alecsandru Patrascu : Added file: http://bugs.python.org/file41137/lto-cpython2-v02.patch ___ Python tracker ___

[issue25702] Link Time Optimizations support for GCC and CLANG

Antoine Pitrou added the comment: Le 23/11/2015 13:18, Alecsandru Patrascu a écrit : > > Do you see it as an configure option (using, for example, an > explicit --with-lto flag) rather than using it automatically? That would be nice. This way people can easily test different combinations of

[issue25702] Link Time Optimizations support for GCC and CLANG

Antoine Pitrou added the comment: Le 23/11/2015 15:44, Alecsandru Patrascu a écrit : > > Meanwhile I've added the patches (v02) for LTO enabled only if the > "./configure --with-lto" command is issued. Cool, thanks! -- ___ Python tracker