[issue25702] Link Time Optimizations support for GCC and CLANG

[Inada Naoki]

Change by Inada Naoki :


--
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Brett Cannon]

Change by Brett Cannon :


--
nosy:  -brett.cannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[David Filiatrault]

Change by David Filiatrault :


--
nosy: +David Filiatrault

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Gregory P. Smith]

Gregory P. Smith added the comment:

the main part of this issue is done but it can't be closed until the 
dependencies listed are also dealt with.  un-assigning myself.

--
assignee: gregory.p.smith -> 
resolution:  -> fixed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Roundup Robot]

Roundup Robot added the comment:

New changeset f710dac07312 by Gregory P. Smith in branch '2.7':
Issue #25702: A --with-lto configure option has been added that will
https://hg.python.org/cpython/rev/f710dac07312

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Gregory P. Smith]

Changes by Gregory P. Smith :


--
dependencies: +test_gdb fails all tests on a profile-opt build configured 
--with-lto

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Gregory P. Smith]

Changes by Gregory P. Smith :


--
dependencies: +test_distutils fails when configured --with-lto

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Gregory P. Smith]

Gregory P. Smith added the comment:

What i committed for 3.5 and 3.6 matches lto-cpython3-v04.patch which just adds 
--with-lto support.  2.7 still needs to be patched.

For reference: Using ubuntu's gcc 5.2.1 i was seeing a 2-3% performance 
increase in the resulting LTO binary vs a plain profile-opt PGO build.  That'll 
vary based on arch and compiler toolchain.

--
assignee:  -> gregory.p.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Roundup Robot]

Roundup Robot added the comment:

New changeset f16ec63055ad by Gregory P. Smith in branch '3.5':
Issue #25702: A --with-lto configure option has been added that will
https://hg.python.org/cpython/rev/f16ec63055ad

New changeset 3103af76f4c4 by Gregory P. Smith in branch 'default':
Issue #25702: A --with-lto configure option has been added that will
https://hg.python.org/cpython/rev/3103af76f4c4

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Alecsandru Patrascu]

Alecsandru Patrascu added the comment:

Maybe an workflow like the one proposed in issue #26359 can be helpful in these 
development phases.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Stefan Krah]

Stefan Krah added the comment:

On Thu, Apr 14, 2016 at 08:55:25AM +, Stefan Krah wrote:
> I use it all the time in development:

... where "it" refers to "./configure && make", not to PGO.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 14.04.2016 10:39, Alecsandru Patrascu wrote:
> 
> @Stefan and @Marc, you say that people don't want to wait for PGO to build 
> when running ./configure && make, but why? Even though many developers use 
> it, this mode is not intended for development, it is production level and 
> should be run once (or at leas a limited number or times), when the 
> developers are sure that everything is fine in the debug mode. As Victor 
> previously said, we should have all the *good* stuff (PGO, LTO, etc) enabled 
> by default, regardless the time needed to do it.

You need to compile Python a lot during Python development and
here the compile speed matters, the performance of the resulting
binary is secondary (as long as it is consistent).

For production, it's easily possible to add those options to configure,
plus it's not 100% clear whether all optimizations really do create
correct code. We've had lots of issues with optimization errors in
compilers in the past and have generally been rather conservative with
the default optimization settings. It's better to have a stable running
Python, than a Python that is fast at failing or creating wrong
results ;-)

I think having these extra options readily accessible and
working is great, and people who know what they are doing can
then use them for the benefit of getting an even faster Python.

Distributors will know what they are doing, so many Python
users will still be able to benefit from them.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Stefan Krah]

Stefan Krah added the comment:

On Thu, Apr 14, 2016 at 08:39:20AM +, Alecsandru Patrascu wrote:
> @Stefan and @Marc, you say that people don't want to wait for PGO to build 
> when running ./configure && make, but why? Even though many developers use 
> it, this mode is not intended for development, it is production level and 
> should be run once (or at leas a limited number or times), when the 
> developers are sure that everything is fine in the debug mode. As Victor 
> previously said, we should have all the *good* stuff (PGO, LTO, etc) enabled 
> by default, regardless the time needed to do it.

I use it all the time in development:

  - For running math tests that would be too slow otherwise.

  - To diagnose invalid accesses that only occur with -O2.

  - To speed up Valgrind runs.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Alecsandru Patrascu]

Alecsandru Patrascu added the comment:

@Stefan and @Marc, you say that people don't want to wait for PGO to build when 
running ./configure && make, but why? Even though many developers use it, this 
mode is not intended for development, it is production level and should be run 
once (or at leas a limited number or times), when the developers are sure that 
everything is fine in the debug mode. As Victor previously said, we should have 
all the *good* stuff (PGO, LTO, etc) enabled by default, regardless the time 
needed to do it.

@Victor, indeed, LTO is not yet good enough to use it stand-alone in CPython. 
That is the reason why it is enabled only with PGO, because applied over it, we 
obtain further speedups than PGO alone. Also Ubuntu uses PGO and LTO in their 
releases.

But in the end maybe `./configure --with-lto && make profile-opt` will have to 
do for everybody.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Antoine Pitrou]

Antoine Pitrou added the comment:

Le 13/04/2016 21:55, Marc-Andre Lemburg a écrit :
>>
>> LTO is not stable on all platforms (according to doko), and people don't
>> want to wait for PGO to build when they just run ./configure && make.
>>
>> --with-pgo and --with-lto is fine.
> 
> Agreed. Let's not make compilation take longer than necessary.
> 
> When doing production builds, people can still enable these
> optimizations as necessary.

Agreed as well. It's enough to make these options sufficiently accessible.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 13.04.2016 21:50, Stefan Krah wrote:
> 
> LTO is not stable on all platforms (according to doko), and people don't
> want to wait for PGO to build when they just run ./configure && make.
> 
> --with-pgo and --with-lto is fine.

Agreed. Let's not make compilation take longer than necessary.

When doing production builds, people can still enable these
optimizations as necessary.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[STINNER Victor]

STINNER Victor added the comment:

> LTO is not stable on all platforms (according to doko), and people don't want 
> to wait for PGO to build when they just run ./configure && make.

Can we have a whitelist of arch known to support PGO and/or LTO? Or maybe a 
blacklist?

Ubuntu already has this knownledge in their package, no?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Stefan Krah]

Stefan Krah added the comment:

LTO is not stable on all platforms (according to doko), and people don't
want to wait for PGO to build when they just run ./configure && make.

--with-pgo and --with-lto is fine.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[STINNER Victor]

STINNER Victor added the comment:

+  --with-lto  Enable Link Time Optimization in PGO builds.
+  Disabled by default.

I don't understand why it's disabled by default. IMHO we must enable all the 
best optimizers options *by default*.

But I expect all optimizations to be disabled by --with-debug.

--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Gregory P. Smith]

Gregory P. Smith added the comment:

Piping up from the peanut gallery here:

If your use case is not doing release builds for production use, i.e. "casual 
use", don't bother with either PGO or LTO.  It won't matter.

Your final build that you Q ship should absolutely use those. (nobody's going 
to disagree with that :)

While I would not reject changes that allow --with-lto to work in the absence 
of PGO, but I don't think it should be anyone's priority.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[INADA Naoki]

INADA Naoki added the comment:

> For sure you are not the only user that has active workloads on the physical 
> machine while you do benchmarks :)

I think largest machine type I chosen (32core) can avoid sharing physical 
machine with other users.

> On the other hand, the path you are going with just LTO is nice for 
> experiments, but for real-world usages is not feasible. Using it in 
> conjunction with PGO is the way to have the best Python interpreter, and I 
> strongly recommend for you to use the v04 versions of the patches.

I agree PGO+LTE is the best.  But I want "only LTO" because:

1) It is a pitfall that `./configure --with-lto && make` doesn't use LTO.
2) PGO makes build too slow.  For casual usecase, I can wait LTO but not PGO.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Alecsandru Patrascu]

Alecsandru Patrascu added the comment:

You are doing measurements on a virtual machine... For sure you are not the 
only user that has active workloads on the physical machine while you do 
benchmarks :)

On the other hand, the path you are going with just LTO is nice for 
experiments, but for real-world usages is not feasible. Using it in conjunction 
with PGO is the way to have the best Python interpreter, and I strongly 
recommend for you to use the v04 versions of the patches.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[INADA Naoki]

INADA Naoki added the comment:

The machine is Google Compute Engine n1-highcpu-32 (Intel Ivy Bridge)

Linux bench 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3 (2016-01-17) 
x86_64 GNU/Linux

cpuinfo:
processor   : 31
vendor_id   : GenuineIntel
cpu family  : 6
model   : 62
model name  : Intel(R) Xeon(R) CPU @ 2.50GHz
stepping: 4
microcode   : 0x1
cpu MHz : 2500.000
cache size  : 30720 KB


command:
$ python perf.py -r -b default ../Python-3.5.1/python-default 
../Python-3.5.1/python-lto

output:
Report on Linux bench 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3 
(2016-01-17) x86_64 
Total CPU cores: 32

### 2to3 ###
Min: 8.692000 -> 8.16: 1.07x faster
Avg: 8.816800 -> 8.253600: 1.07x faster
Significant (t=8.07)
Stddev: 0.12726 -> 0.09027: 1.4098x smaller

### chameleon_v2 ###
Min: 6.756928 -> 6.414046: 1.05x faster
Avg: 6.849192 -> 6.666536: 1.03x faster
Significant (t=20.88)
Stddev: 0.04413 -> 0.07555: 1.7120x larger

### fastpickle ###
Min: 0.540906 -> 0.564253: 1.04x slower
Avg: 0.549624 -> 0.579263: 1.05x slower
Significant (t=-34.29)
Stddev: 0.00427 -> 0.00752: 1.7622x larger

### nbody ###
Min: 0.260169 -> 0.273837: 1.05x slower
Avg: 0.267334 -> 0.280441: 1.05x slower
Significant (t=-34.05)
Stddev: 0.00257 -> 0.00286: 1.1125x larger

### regex_v8 ###
Min: 0.047335 -> 0.044750: 1.06x faster
Avg: 0.049424 -> 0.046788: 1.06x faster
Significant (t=10.46)
Stddev: 0.00174 -> 0.00182: 1.0469x larger

The following not significant results are hidden, use -v to show them:
django_v3, fastunpickle, json_dump_v2, json_load, tornado_http.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Alecsandru Patrascu]

Alecsandru Patrascu added the comment:

>From our experience, pybench only is not a representative benchmark. Instead, 
>if you like to measure performance close to real workloads, you can run the 
>Grand Unified Python Benchmark suite, that is more complete. 

Also, you need to take into consideration the hardware and software 
environment. For this, you can read the initial comment at this issue, section 
"Hardware and OS Configuration", to see the approach we have here at Intel.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[INADA Naoki]

INADA Naoki added the comment:

I've tried LTO without PGO in Debian Jessie.

$ LTOFLAGS='-flto -fuse-linker-plugin -ffat-lto-objects -flto-partition=none'
$ CFLAGS=$LTOFLAGS LDFLAGS=$LTOFLAGS ./configure --prefix=...
$ make -j32

results is here (compared with neither LTO and PGO):

Test minimum run-timeaverage  run-time
 thisother   diffthisother   diff
---
  BuiltinFunctionCalls:47ms50ms   -6.6%48ms51ms   -6.0%
   BuiltinMethodLookup:29ms29ms   -1.3%29ms29ms   -0.1%
 CompareFloats:32ms33ms   -2.8%34ms34ms   -0.5%
 CompareFloatsIntegers:67ms70ms   -3.9%69ms71ms   -3.1%
   CompareIntegers:48ms46ms   +5.1%49ms47ms   +5.8%
CompareInternedStrings:30ms31ms   -1.9%31ms31ms   -1.6%
  CompareLongs:28ms26ms   +8.0%29ms27ms   +8.5%
CompareStrings:26ms26ms   -0.9%27ms26ms   +1.5%
ComplexPythonFunctionCalls:47ms51ms   -8.9%48ms52ms   -7.8%
 ConcatStrings:32ms33ms   -3.2%33ms34ms   -2.2%
   CreateInstances:51ms52ms   -2.5%52ms53ms   -3.5%
CreateNewInstances:38ms40ms   -4.5%39ms41ms   -4.4%
   CreateStringsWithConcat:68ms69ms   -1.4%70ms71ms   -0.4%
  DictCreation:53ms51ms   +5.2%55ms52ms   +6.7%
 DictWithFloatKeys:41ms42ms   -2.2%43ms43ms   -0.0%
   DictWithIntegerKeys:34ms34ms   +0.1%35ms35ms   +0.5%
DictWithStringKeys:31ms32ms   -1.3%32ms32ms   -1.6%
  ForLoops:26ms30ms  -12.1%28ms30ms   -8.7%
IfThenElse:42ms41ms   +2.6%43ms41ms   +5.0%
   ListSlicing:40ms40ms   -0.8%41ms41ms   -0.4%
NestedForLoops:42ms42ms   -0.3%43ms43ms   +0.6%
  NestedListComprehensions:42ms47ms  -11.9%45ms50ms  -10.5%
  NormalClassAttribute:89ms96ms   -7.9%92ms98ms   -5.9%
   NormalInstanceAttribute:47ms45ms   +4.8%48ms45ms   +4.9%
   PythonFunctionCalls:41ms44ms   -7.5%41ms45ms   -7.4%
 PythonMethodCalls:53ms59ms   -9.4%55ms60ms   -8.5%
 Recursion:69ms73ms   -5.1%71ms74ms   -4.2%
  SecondImport:36ms41ms  -12.0%38ms42ms   -9.9%
   SecondPackageImport:45ms42ms   +6.5%46ms43ms   +7.0%
 SecondSubmoduleImport:   115ms   107ms   +7.9%   117ms   108ms   +7.9%
   SimpleComplexArithmetic:27ms29ms   -6.5%28ms30ms   -4.5%
SimpleDictManipulation:60ms65ms   -7.8%61ms66ms   -7.0%
 SimpleFloatArithmetic:33ms30ms   +7.4%34ms31ms   +8.3%
  SimpleIntFloatArithmetic:36ms38ms   -3.3%37ms38ms   -4.0%
   SimpleIntegerArithmetic:36ms38ms   -5.2%37ms38ms   -4.1%
  SimpleListComprehensions:36ms37ms   -3.2%38ms41ms   -7.5%
SimpleListManipulation:34ms34ms   -1.3%35ms38ms   -6.8%
  SimpleLongArithmetic:26ms26ms   +0.3%27ms30ms   -7.5%
SmallLists:45ms47ms   -4.1%46ms56ms  -17.2%
   SmallTuples:51ms54ms   -6.3%53ms62ms  -14.8%
 SpecialClassAttribute:92ms97ms   -5.0%95ms99ms   -4.8%
  SpecialInstanceAttribute:46ms45ms   +2.5%48ms46ms   +3.9%
StringMappings:71ms   100ms  -29.0%73ms   101ms  -27.8%
  StringPredicates:49ms59ms  -17.8%50ms60ms  -16.5%
 StringSlicing:48ms47ms   +3.3%79ms47ms  +66.2%
 TryExcept:24ms29ms  -16.9%25ms30ms  -15.8%
TryFinally:35ms37ms   -6.0%36ms38ms   -4.6%
TryRaiseExcept:12ms13ms   -7.5%13ms14ms   -7.2%
  TupleSlicing:48ms50ms   -2.9%49ms51ms   -2.7%
   WithFinally:52ms57ms   -8.4%53ms58ms   -8.2%
   WithRaiseExcept:42ms46ms   -8.8%43ms47ms   -9.1%
---
Totals:  2291ms  2398ms   -4.5%  2390ms  2470ms   -3.2%

(this=lto.pybench, other=default.pybench)

--

___
Python tracker 

___

[issue25702] Link Time Optimizations support for GCC and CLANG

[Alecsandru Patrascu]

Alecsandru Patrascu added the comment:

I understand now your question. LTO is not enabled when running just `make`, 
only in `make profile-opt`

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[INADA Naoki]

INADA Naoki added the comment:

Sorry my poor English.
I meant that "Does `./configure --with-lto && make` use LTO?".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Alecsandru Patrascu]

Alecsandru Patrascu added the comment:

Yes, you can use LTO without PGO, but the proposed ways it's more efficient and 
makes more sense for CPython builds.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Alecsandru Patrascu]

Alecsandru Patrascu added the comment:

As Steve mentioned, the Microsoft compiler uses LTO (they call it Link-Time 
Code Generation) and the flags are used when compiling CPython on Windows 
systems. Thus our proposal to enable it on GCC and CLANG also.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Steve Dower]

Steve Dower added the comment:

MSVC has had Link-Time Code Generation for many releases, and it should have 
been used for all 2.7 releases (definitely used in 3.5+) to optimize references 
between object files. I assume this is equivalent to LTO.

We currently don't use PGO in the official Windows builds, but it is a 
supported build configuration.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Brett Cannon]

Brett Cannon added the comment:

To help answer MAL's question about Windows: I know the python.org installers 
are **not** built with PGO, but I don't know about LTO.

--
nosy: +steve.dower

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[STINNER Victor]

STINNER Victor added the comment:

I suggest to only modify the default branch and work with downstream (like
Linux vendors) to compile Python with best compiler options.

I'm talking about the default compilation mode. Maybe we can add a
configure option to 2.7 and 3.5, disabled by default, to use best options.

Sorry I didn't read the whole discussion.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Alecsandru Patrascu]

Changes by Alecsandru Patrascu :


Added file: http://bugs.python.org/file41666/lto-cpython3-v04.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Alecsandru Patrascu]

Alecsandru Patrascu added the comment:

Thank you for your feedback, I've updated the patches and now LTO flags are 
used only when building with PGO (v04). CFLAGS/LDFLAGS remain untouched, as 
Antoine and Victor suggested is better.

--
Added file: http://bugs.python.org/file41665/lto-cpython2-v04.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 20.01.2016 09:37, Antoine Pitrou wrote:
> Let me add that downstream distributors already customize compilation options 
> (Ubuntu's Python is compiled with both PGO and LTO enabled, AFAIR), so this 
> change may only really affect the tiny subset of non-Windows users that 
> compile Python themselves.

Are the Windows installers on python.org compiled with PGO and
LTO enabled ?

If not, then the patch would also effect the not-so-tiny fraction
of Python users on Windows ;-)

BTW: It may make sense to start collecting the various performance
related optional patches to Python 2.7 on a wiki page for interested
parties to use.

--
nosy: +lemburg

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Antoine Pitrou]

Antoine Pitrou added the comment:

> If not, then the patch would also effect the not-so-tiny fraction
of Python users on Windows ;-)

I don't see how enabling LTO for gcc and clang could ever affect our Windows 
users ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 20.01.2016 11:38, Antoine Pitrou wrote:
>> If not, then the patch would also effect the not-so-tiny fraction
>> of Python users on Windows ;-)
> 
> I don't see how enabling LTO for gcc and clang could ever affect our Windows 
> users ;-)

You have a point there, but perhaps we could start offering
an ICC compiled version for Windows ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[STINNER Victor]

Changes by STINNER Victor :


--
nosy:  -haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[R. David Murray]

R. David Murray added the comment:

My understanding is that we (starting with Guido) have made a blanket exception 
for 2.7 for useful performance and build-system-only related patches.  That 
doesn't mean *anything* can go in (the usual rules about "is this worth 
it/backward compatible/won't break things" still apply) but it is a lower bar 
than is true for other maintenance only releases.  Perhaps my understanding is 
in error, though.  

I believe Intel is committed to supporting this, so if there do turn out to be 
any maintenance issues they can handle them.  (Which IIUC is Nick's argument: 
if someone wants to support 2.7 with stuff we are willing to let in, we should 
let them as long as they credibly commit to supporting it.)  I'm currently part 
of that Intel support, though, so someone else should rule on this.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Antoine Pitrou]

Antoine Pitrou added the comment:

> Unfortunately, 2.7 doesn't have even CFLAGS_NODIST; I suspect it may be time 
> to backport that.

I don't think now is a good time to introduce instability in the 2.7 branch.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Zachary Ware]

Zachary Ware added the comment:

I'm a bit concerned that the flags are being added unconditionally to CFLAGS 
and LDFLAGS (when configured --with-lto), which means extensions are forced 
into it as well.  I think it would be better to use CFLAGS_NODIST and to add 
LDFLAGS_NODIST.  Unfortunately, 2.7 doesn't have even CFLAGS_NODIST; I suspect 
it may be time to backport that.

--
nosy: +zach.ware

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[R. David Murray]

Changes by R. David Murray :


--
nosy: +r.david.murray

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Zachary Ware]

Zachary Ware added the comment:

Unless I'm just missing something, I don't see how introducing CFLAGS_NODIST 
and LDFLAGS_NODIST to 2.7 would introduce instability.  It should be a fairly 
non-invasive change, restricted to configure and the Makefile; both vars should 
usually be empty and thus builds should be entirely unaffected unless options 
like --with-lto are chosen.


On a separate note about the patch: as mentioned in msg251305, it's probably 
better to restrict adding the LTO flags to just the profile-opt targets, even 
with the --with-lto check.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Alecsandru Patrascu]

Alecsandru Patrascu added the comment:

Hello, I've added an updated set of patches (v03) for the current CPython2 and 
CPython3 codebase. Also made some small changes to reduce the number of places 
where the flags are set.

--
Added file: http://bugs.python.org/file41493/lto-cpython2-v03.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Alecsandru Patrascu]

Changes by Alecsandru Patrascu :


Added file: http://bugs.python.org/file41494/lto-cpython3-v03.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Alecsandru Patrascu]

Alecsandru Patrascu added the comment:

I'm adding Brett, Gregory, Stefan and Victor as nosy because this issue might 
be interesting for them also.

--
nosy: +brett.cannon, gregory.p.smith, haypo, scoder, skrah

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Antoine Pitrou]

Antoine Pitrou added the comment:

LTO only exists on recent versions of gcc, so the configure script should 
probably do some version checking.

Also we can't enable it by default as 1) it makes compile times much longer 2) 
there are some bugs in some gcc versions (see e.g. 
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=753134).

--
nosy: +pitrou
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Alecsandru Patrascu]

Alecsandru Patrascu added the comment:

LTO exists in GCC since version 4.5, but it is true that only recent versions 
(>=4.8) perform it in good conditions. It is not enabled by default in this 
patch, it is only available when building with PGO support. Running just "make" 
will not activate the LTO flags.

Do you see it as an configure option (using, for example, an explicit 
--with-lto flag) rather than using it automatically?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Alecsandru Patrascu]

New submission from Alecsandru Patrascu:

Title: Link Time Optimizations support for GCC and CLANG

Hi All,

This is Alecsandru from Server Scripting Languages Optimization team at Intel 
Corporation. I would like to submit a patch that adds support for Link Time 
Optimization (LTO) when using GCC and CLANG to compile CPython2 and CPython3. 
LTO is a compiler assisted optimization technique that is performed by the 
compiler at link time.

Combined with Profile Guided Optimization (PGO), enabled when running "make 
profile-opt", and running the Grand Unified Python Benchmark (GUPB), a speedup 
up to 11%, with a few regressions, was observed comparing with PGO only. 
Compared with a default build, a performance gain as high as 26% was observed 
from PGO+LTO. In addition, we are also seeing 2% boost in throughput rate from 
our OpenStack Swift setup comparing with PGO only. Our GUPB performance 
evaluation was conducted on Intel SkyLake/Broadwell systems running 
CentOS/Ubuntu, with CLANG/LLVM and GCC 4.*/5.*. Our OpenStack Swift performance 
was done on various systems consisting of XEON and Avoton processors.

Steps:
==

1. Get the CPython source codes
hg clone https://hg.python.org/cpython cpython
cd cpython
hg update 2.7 (for CPython2)

2. Build the binary
a) Default:
./configure
make

b) PGO:
./configure
make profile-opt

c) PGO+LTO:
Copy the attached patch files
hg import --no-commit lto-cpython3-v01.patch (for CPython3)
hg import --no-commit lto-cpython2-v01.patch (for CPython2)
./configure
make profile-opt


Hardware and OS Configuration
=
Hardware:   Intel XEON (Broadwell-DE) 8 Cores

BIOS settings:  Intel Turbo Boost Technology: false
Hyper-Threading: false  

OS: Ubuntu 14.04.3 LTS Server

OS configuration:   Address Space Layout Randomization (ASLR) disabled to 
reduce run
to run variation by echo 0 > 
/proc/sys/kernel/randomize_va_space
CPU frequency set fixed at 2.6GHz

GCC version:GCC version 4.9.2

Benchmark:  Grand Unified Python Benchmark from 
https://hg.python.org/benchmarks/


Measurements and Results

A. Repository:
GUPB Benchmark:
hg id :  2979f5ce6a0c tip
hg --debug id -i : 2979f5ce6a0cee994d5485401945d8457bb0afac

CPython3:
hg id : 21a28f6de358
hg id -r 'ancestors(.) and tag()': 374f501f4567 (3.5) v3.5.0
hg --debug id -i : 21a28f6de3582833652c958b8fd6ae8448b61c7c

CPython2:
hg id : a37ea1d56e98 (2.7)
hg id -r 'ancestors(.) and tag()': 15c95b7d81dc (2.7) v2.7.10
hg --debug id -i : a37ea1d56e98eb158750d3e495a5cf524e8c3980


B. Results: 
CPython2 and CPython3 sample results, measured on a Broadwell platform, can be 
viewed in Table 1 and 2. On the first column (Benchmark) you can see the 
benchmark name, on the second (%D) the speedup compared with the default 
version and on the third column (%PGO) the speedup compared with just PGO; a 
higher value is better.

Table 1. CPython2 results:
Benchmark   %D  %PGO

raytrace18  3
chaos   16  5
django_v2   16  6
mako16  6
pathlib 15  3
simple_logging  15  1
slowpickle  15  5
django  14  4
go  14  4
richards13  -1
float   12  4
slowunpickle12  4
etree_process   11  3
fastunpickle11  6
formatted_logging   11  3
nqueens 11  1
regex_compile   11  3
etree_iterparse 10  4
mako_v2 10  3
telco   10  5
pybench 9   1
hexiom2 9   1
html5lib_warmup 9   3
meteor_contest  9   4
pickle_list 9   5
2to38   2
bzr_startup 8   2
chameleon   8   0
etree_generate  8   2
regex_v88   3
silent_logging  8   1
fannkuch7   1
html5lib7   3
json_load   7   -5
tornado_http7   3
call_method_slots   6   3
json_dump_v26   -4
spambayes   6   2
unpickle_list   6   0
etree_parse 5   3
fastpickle  5   4
rietveld5   1
call_method 4   -1
normal_startup  4   2
startup_nosite  4   2
slowspitfire3   0
ssbench 4   2
call_method_unknown 1   -6
json_dump   1   -4
nbody   1   1
pidigits1   -10
pickle_dict 0   -1
regex_effbot0   -2
spectral_norm   0   -3
call_simple -3  -3

[issue25702] Link Time Optimizations support for GCC and CLANG

[Alecsandru Patrascu]

Changes by Alecsandru Patrascu :


Added file: http://bugs.python.org/file41131/lto-cpython3-v01.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Alecsandru Patrascu]

Changes by Alecsandru Patrascu :


--
keywords: +patch
Added file: http://bugs.python.org/file41130/lto-cpython2-v01.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Alecsandru Patrascu]

Changes by Alecsandru Patrascu :


Added file: http://bugs.python.org/file41138/lto-cpython3-v02.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Alecsandru Patrascu]

Alecsandru Patrascu added the comment:

Meanwhile I've added the patches (v02) for LTO enabled only if the "./configure 
--with-lto" command is issued.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Alecsandru Patrascu]

Changes by Alecsandru Patrascu :


Added file: http://bugs.python.org/file41137/lto-cpython2-v02.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Antoine Pitrou]

Antoine Pitrou added the comment:

Le 23/11/2015 13:18, Alecsandru Patrascu a écrit :
> 
> Do you see it as an configure option (using, for example, an
> explicit --with-lto flag) rather than using it automatically?

That would be nice. This way people can easily test different
combinations of flags.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25702] Link Time Optimizations support for GCC and CLANG

[Antoine Pitrou]

Antoine Pitrou added the comment:

Le 23/11/2015 15:44, Alecsandru Patrascu a écrit :
> 
> Meanwhile I've added the patches (v02) for LTO enabled only if the 
> "./configure --with-lto" command is issued.

Cool, thanks!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com