[issue21122] CPython fails to build modules with LLVM LTO on Mac OS X

[Brett Cannon]

Changes by Brett Cannon :


--
resolution:  -> fixed
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21122] CPython fails to build modules with LLVM LTO on Mac OS X

[Roundup Robot]

Roundup Robot added the comment:

New changeset cc5f8179a7ba by Ned Deily in branch 'default':
Issue #21122: Fix LTO builds on OS X.
https://hg.python.org/cpython/rev/cc5f8179a7ba

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21122] CPython fails to build modules with LLVM LTO on Mac OS X

[Brett Cannon]

Brett Cannon added the comment:

Here is a patch to turn on `-Wl,-export_dynamic` when building with LTO. 
Unfortunately I have a bunch of tests that fail when running with LTO+PGO of 
the form of:

[ 95/398] test_bytes
Traceback (most recent call last):
  File "/Users/brettcannon/Repositories/python/cpython/3.5/Lib/runpy.py", line 
193, in _run_module_as_main
"__main__", mod_spec)
  File "/Users/brettcannon/Repositories/python/cpython/3.5/Lib/runpy.py", line 
85, in _run_code
exec(code, run_globals)
  File 
"/Users/brettcannon/Repositories/python/cpython/3.5/Lib/test/__main__.py", line 
3, in 
regrtest.main_in_temp_cwd()
  File 
"/Users/brettcannon/Repositories/python/cpython/3.5/Lib/test/regrtest.py", line 
1593, in main_in_temp_cwd
main()
  File 
"/Users/brettcannon/Repositories/python/cpython/3.5/Lib/test/regrtest.py", line 
756, in main
raise Exception("Child error on {}: {}".format(test, result[1]))
Exception: Child error on test_bytes: Exit code -6

--
keywords: +patch
resolution: fixed -> 
status: closed -> open
Added file: http://bugs.python.org/file44352/issue21122.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21122] CPython fails to build modules with LLVM LTO on Mac OS X

[Brett Cannon]

Changes by Brett Cannon :


--
components: +Macintosh

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21122] CPython fails to build modules with LLVM LTO on Mac OS X

[Jonas Wagner]

Jonas Wagner added the comment:

No response for a while, and problem solved... closing.

--
resolution:  - fixed
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21122
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21122] CPython fails to build modules with LLVM LTO on Mac OS X

[Jonas Wagner]

Jonas Wagner added the comment:

I confirm that this also works with my self-compiled Clang 3.4. -export_dynamic 
was the missing option.

Is a good place to document this?

Otherwise, I think this issue can be closed. Thanks a lot for the help!

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21122
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21122] CPython fails to build modules with LLVM LTO on Mac OS X

[Jonas Wagner]

Jonas Wagner added the comment:

Thanks Ned, this is interesting!

I don't know about Mac OS, but on Ubuntu, LTO and PGO apparently make Python 
around 10% faster (see #17781). However, that data point refers to GCC's LTO, 
not LLVM's.

Personally I'm interested in LTO because I want to obtain whole-program LLVM 
bitcode files (for use in a research project about instrumentation). However, 
if there is something I can help (e.g., running benchmarks with different 
compilation settings), let me know.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21122
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21122] CPython fails to build modules with LLVM LTO on Mac OS X

[Ronald Oussoren]

Ronald Oussoren added the comment:

I've used -O4 for extensions in the past (which until recently implied LTO) and 
that worked fine. 

I'm pretty sure that I haven't used LTO for python itself, apart from a some 
tests with an early version llvm-gcc where using LTO for building python used 
to crash the compiler :-)

BTW. There's no clear pattern in the missing symbols. The missing symbols for 
global functions could be due to aggressive inlining (and then deciding that 
the standalone function isn't needed anymore), but that is fairly unlikely and 
wouldn't explain the missing data symbols.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21122
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21122] CPython fails to build modules with LLVM LTO on Mac OS X

[Ronald Oussoren]

Ronald Oussoren added the comment:

Have you tried the -export_dynamic option for ld(1):


-export_dynamic
 Preserves all global symbols in main executables during LTO.  
Without this option, Link Time Optimization is allowed to inline and remove 
global functions. This
 option is used when a main executable may load a plug-in which 
requires certain symbols from the main executable.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21122
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21122] CPython fails to build modules with LLVM LTO on Mac OS X

[Ronald Oussoren]

Ronald Oussoren added the comment:

This works for me (with a separate build directory):

  CC=clang CXX=clang++ CFLAGS=-g -flto LDFLAGS=-flto -Wl,-export_dynamic 
../configure

This is on OSX 10.9.2, with Xcode 5.1, and clang --version says:

$ clang --version
Apple LLVM version 5.1 (clang-503.0.38) (based on LLVM 3.4svn)
Target: x86_64-apple-darwin13.1.0
Thread model: posix

Tests are still running, but so far there are no unexpected failures.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21122
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21122] CPython fails to build modules with LLVM LTO on Mac OS

[Jonas Wagner]

New submission from Jonas Wagner:

CPython fails to build with LLVM's link-time optimization (LTO) in Mac OS. Very 
similar commands work on Linux.

I'm currently configuring CPython as follows:

on Linux:
RANLIB=ar -s --plugin=/path/to/llvm/lib/LLVMgold.so 
CC=/path/to/llvm/bin/clang CXX=/path/to/llvm/bin/clang++ CFLAGS=-g -flto 
LDFLAGS=-flto ./configure

on Mac OS:
CC=/path/to/llvm/bin/clang CXX=/path/to/llvm/bin/clang++ CFLAGS=-g -flto 
LDFLAGS=-flto ./configure

The RANLIB variable should not be needed on Mac, because its toolchain does not 
need a GOLD plugin.

On Linux, the above builds correctly and passes most of the test suite. On Mac, 
I receive the following error (similar for other extensions):

building '_scproxy' extension
/usr/bin/clang -Wno-unused-result -Werror=declaration-after-statement -DNDEBUG 
-g -fwrapv -O3 -Wall -Wstrict-prototypes -g -flto -I../cpython/Include -I. 
-I/usr/local/include -I../cpython/Include -I../cpython-lto-build -c 
../cpython/Modules/_scproxy.c -o 
build/temp.macosx-10.9-x86_64-3.5/path/to/cpython/Modules/_scproxy.o
/usr/bin/clang -bundle -undefined dynamic_lookup -flto 
build/temp.macosx-10.9-x86_64-3.5/path/to/cpython/Modules/_scproxy.o 
-L/usr/local/lib -o build/lib.macosx-10.9-x86_64-3.5/_scproxy.so -framework 
SystemConfiguration -framework CoreFoundation
*** WARNING: renaming _scproxy since importing it failed: 
dlopen(build/lib.macosx-10.9-x86_64-3.5/_scproxy.so, 2): Symbol not found: 
__Py_NoneStruct
  Referenced from: build/lib.macosx-10.9-x86_64-3.5/_scproxy.so
  Expected in: flat namespace
 in build/lib.macosx-10.9-x86_64-3.5/_scproxy.so

--
components: Extension Modules
messages: 215307
nosy: Sjlver
priority: normal
severity: normal
status: open
title: CPython fails to build modules with LLVM LTO on Mac OS
type: compile error
versions: Python 3.4

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21122
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21122] CPython fails to build modules with LLVM LTO on Mac OS

[STINNER Victor]

Changes by STINNER Victor victor.stin...@gmail.com:


--
nosy: +haypo

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21122
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21122] CPython fails to build modules with LLVM LTO on Mac OS

[STINNER Victor]

Changes by STINNER Victor victor.stin...@gmail.com:


--
nosy: +koobs

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21122
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21122] CPython fails to build modules with LLVM LTO on Mac OS

[Jonas Wagner]

Jonas Wagner added the comment:

I am indeed using Clang 3.4 (both the one that ships with Mac OS, and a version 
compiled from the sources).

However, the errors I get are rather different than #20767. In particular, 
Clang finishes successfully and does produce shared object files; they just 
don't seem to be loadable.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21122
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21122] CPython fails to build modules with LLVM LTO on Mac OS

[STINNER Victor]

STINNER Victor added the comment:

What is your clang version? See also issue #20767.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21122
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21122] CPython fails to build modules with LLVM LTO on Mac OS X

[Ned Deily]

Ned Deily added the comment:

Just as an experiment (using the 3.4 branch and the Xcode 5.1 clang), the list 
of unique symbols not found during the test dlopen in setup.py when using -flto:

 _PyArg_ParseTuple
 _PyArg_ParseTupleAndKeywords
 _PyBaseObject_Type
 _PyBool_Type
 _PyByteArray_Type
 _PyBytes_Type
 _PyCFunction_Type
 _PyExc_AssertionError
 _PyExc_BufferError
 _PyExc_EOFError
 _PyExc_IndexError
 _PyExc_IOError
 _PyExc_KeyError
 _PyExc_LookupError
 _PyExc_MemoryError
 _PyExc_NotImplementedError
 _PyExc_OSError
 _PyExc_OverflowError
 _PyExc_RuntimeError
 _PyExc_TypeError
 _PyExc_ValueError
 _PyModule_Create2
 __Py_NoneStruct

Anyone see a pattern there? 

Do we know if anyone has tried to use LTO with a Python build previously?  I've 
never tried it myself and there certainly could be ld and/or dyld differences 
on OS X.  Also, some thought would need to go into and tests developed to see 
what the performance trade-offs are.  For example, I could imagine that LTO 
might be have more impact if the standard library extension modules were 
statically linked, e.g. via Modules/Setup*.  And there are at least three 
separate current build configurations to consider on OS X: unshared, 
--enable-shared, --enable-framework.  One would need to look at things like 
what effect these all have on memory and shared memory footprints as well as 
cpu resources and real time, with and without LTO and/or other optimizations.  
It certainly would be an interesting project for someone with the interest and 
time.

Potentially supporting LTO seems to me to be more of a feature than a bug so I 
think should be considered a 3.5 issue, at least initially.

--
components: +Build -Extension Modules
nosy: +brett.cannon, ned.deily, ronaldoussoren
title: CPython fails to build modules with LLVM LTO on Mac OS - CPython fails 
to build modules with LLVM LTO on Mac OS X
type: compile error - 
versions: +Python 3.5 -Python 3.4

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21122
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com