[issue29439] _decimal on Android requires linking with libm

[Chi Hsuan Yen]

Chi Hsuan Yen added the comment:

Actually my device is 6.0. Seems there's nothing interesting between API 23 and 
24 on android-changes-for-ndk-developers.md :)

Anyway, _decimal should not depend on the whether the python binary references 
to libm.so or not. Instead, _decimal should explicitly have an explicit 
DT_NEEDED entry to libm.so if it uses symbols from libm.

> Strange that on other systems the compilers don't complain (usually they do).

I guess on most system python is built with --enable-shared, so _decimal.so has 
an entry to libpython3.7m.so, and the latter references libm.so. If a linker 
supports recursive search, it can find the symbol.

For future references, here's the partial output of logcat when I run python 
with `LD_DEBUG=3 python3.7m` and import _decimal:

02-07 19:43:25.697  1026610266 linker  D  DEBUG: 
/data/local/tmp/python3/usr/lib/python3.7/lib-dynload/_decimal.cpython-37m.so: 
looking up log10 in python3.7m (from global group)
02-07 19:43:25.697  1026610266 linker  I  SEARCH log10 in 
python3.7m@0x55634f3000 h=735a40(elf) 452
02-07 19:43:25.697  1026610266 linker  I  NOT FOUND log10 
in python3.7m@0x55634f3000 735a40 452
02-07 19:43:25.698  1026610266 linker  D  DEBUG: 
/data/local/tmp/python3/usr/lib/python3.7/lib-dynload/_decimal.cpython-37m.so: 
looking up log10 in /system/vendor/lib64/libNimsWrap.so (from global group)
02-07 19:43:25.698  1026610266 linker  I  SEARCH log10 in 
/system/vendor/lib64/libNimsWrap.so@0x7f7c5fb000 (gnu)
02-07 19:43:25.698  1026610266 linker  I  NOT FOUND log10 
in /system/vendor/lib64/libNimsWrap.so@0x7f7c5fb000
02-07 19:43:25.698  1026610266 linker  D  DEBUG: 
/data/local/tmp/python3/usr/lib/python3.7/lib-dynload/_decimal.cpython-37m.so: 
looking up log10 in 
/data/local/tmp/python3/usr/lib/python3.7/lib-dynload/_decimal.cpython-37m.so 
(from local group)
02-07 19:43:25.698  1026610266 linker  I  SEARCH log10 in 
/data/local/tmp/python3/usr/lib/python3.7/lib-dynload/_decimal.cpython-37m.so@0x7f7c157000
 h=735a40(elf) 49
02-07 19:43:25.698  1026610266 linker  I  NOT FOUND log10 
in 
/data/local/tmp/python3/usr/lib/python3.7/lib-dynload/_decimal.cpython-37m.so@0x7f7c157000
 735a40 49
02-07 19:43:25.698  1026610266 linker  D  DEBUG: 
/data/local/tmp/python3/usr/lib/python3.7/lib-dynload/_decimal.cpython-37m.so: 
looking up log10 in libdl.so (from local group)
02-07 19:43:25.698  1026610266 linker  I  SEARCH log10 in 
libdl.so@0x0 h=735a40(elf) 0
02-07 19:43:25.698  1026610266 linker  I  NOT FOUND log10 
in libdl.so@0x0 735a40 0
02-07 19:43:25.698  1026610266 linker  D  DEBUG: 
/data/local/tmp/python3/usr/lib/python3.7/lib-dynload/_decimal.cpython-37m.so: 
looking up log10 in /system/lib64/libc.so (from local group)
02-07 19:43:25.698  1026610266 linker  I  SEARCH log10 in 
/system/lib64/libc.so@0x7f7c4f6000 (gnu)
02-07 19:43:25.698  1026610266 linker  I  NOT FOUND log10 
in /system/lib64/libc.so@0x7f7c4f6000
02-07 19:43:25.698  1026610266 linker  D  DEBUG: cannot 
locate symbol "log10" referenced by 
"/data/local/tmp/python3/usr/lib/python3.7/lib-dynload/_decimal.cpython-37m.so"...

And BioniC's linker only checks HASH table of ELF: 
https://android.googlesource.com/platform/bionic/+/master/linker/linker_soinfo.cpp#277,
 so log10 is not found in python3.7m

> This is misleading

Maybe it is. I mean "I didn't write a patch for _decimal at issue21668 as 
_decimal didn't build then"

> Perhaps test_decimal should fail for CPython if _decimal can't be imported.

I create a simple script to ensure all installed dynamic modules can be 
imported: 
https://github.com/yan12125/python3-android/blob/master/devscripts/import_all.py.
 It may be worth to transform it into a proper unittest and put it into 
Lib/test/

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29439] _decimal on Android requires linking with libm

[Xavier de Gaye]

Xavier de Gaye added the comment:

> Perhaps test_decimal should fail for CPython if _decimal can't be imported.
This is not a problem specific to _decimal, the same problem exists for 
third-party extension modules, for example pyephem 
(https://github.com/brandon-rhodes/pyephem/issues/114) and existed with some 
other Python extension modules (issue 21668).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29439] _decimal on Android requires linking with libm

[Xavier de Gaye]

Xavier de Gaye added the comment:

With the following module named dlsym.py, the command 'python dlsym.py log10' 
produces the same successful output on linux and on Android API 21 (when 
_decimal is not explicitly linked with libm.so, i.e.without changeset 
b60b46ad8751):
  'The symbol "log10" is resolved.'

-- dlsym.py ---
import sys, os
from ctypes import *

if len(sys.argv) != 2:
sys.exit('Error: the symbol name is required.')
symbol = sys.argv[1]
libdl = CDLL('libdl.so')
libdl.dlopen.restype = c_void_p
libdl.dlsym.restype = c_void_p
hdle = libdl.dlopen(None, os.RTLD_NOW | RTLD_GLOBAL)
if hdle is None:
print('Cannot get a handle on the global symbol object.')
else:
v = libdl.dlsym(c_void_p(hdle), symbol.encode())
_not = 'not ' if v is None else ''
print('The symbol "%s" is %sresolved.' % (symbol, _not))
---

This is as expected since the python executable is linked with the math 
library. However on Android API 21 importing _decimal fails with the error 
message reported earlier:
  'dlopen failed: cannot locate symbol "log10"'.

So this seems to be a bug with Android dlopen() at API 21 since the 'log10' 
symbol does exist in the process image relocation tables as shown by dlsym.py. 
This is confirmed by the fact that the import does not fail anymore at API 24 
(still without changeset b60b46ad8751). This change of behavior may be a side 
effect of the changes reported in this bionic (Android libc) changelog entry:

https://android.googlesource.com/platform/bionic/+show/refs/heads/master/android-changes-for-ndk-developers.md#36

For completness, my (possibly wrong) interpretation of why changeset 
b60b46ad8751 fixes the problem at API 21:
Changeset b60b46ad8751 adds the '-lm' command line option to the linker and 
this adds [1] a new DT_NEEDED entry to the .dynamic section of the shared 
library ELF file as shown by the command 'readelf -d 
build/lib.linux-x86_64-3.7-pydebug/_decimal.cpython-37dm-x86_64-linux-gnu.so':

Dynamic section at offset 0x54d30 contains 26 entries:
  TagType Name/Value
 0x0001 (NEEDED) Shared library: [libm.so.6]
 0x0001 (NEEDED) Shared library: [libpthread.so.0]
 0x0001 (NEEDED) Shared library: [libc.so.6]
 ...

The Android loader lookup for 'log10' is now successful at API 21 and the 
symbol is found in the libm.so.6 library listed in the DT_NEEDED lists of 
needed libraries.

[1] See option '--as-needed' in the 'ld' man pages or see the ELF 
specification. Note that the Android toolchains use the clang compiler but 
continue to use the GNU binutils tools including the linker 'ld'.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29439] _decimal on Android requires linking with libm

[Stefan Krah]

Stefan Krah added the comment:

Perhaps test_decimal should fail for CPython if _decimal can't be imported.

All compilers that I tested seem to link fine without -lm in this case.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29439] _decimal on Android requires linking with libm

[Xavier de Gaye]

Xavier de Gaye added the comment:

> This wasn't fixed because _decimal didn't build before issue26846 lands.

This is misleading, the problem went unnoticed when not linking with '-lm' 
because:
* The _decimal extension module builds without any warning.
* test_decimal does not fail.

One way to demonstrate the problem in the interactive interpreter on Android:
>>> import decimal
>>> import _decimal
Traceback (most recent call last):
  File "", line 1, in 
ImportError: dlopen failed: cannot locate symbol "log10" referenced by 
"_decimal.cpython-37dm.so"...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29439] _decimal on Android requires linking with libm

[Stefan Krah]

Changes by Stefan Krah :


--
resolution:  -> fixed
stage:  -> resolved
status: open -> closed
versions: +Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29439] _decimal on Android requires linking with libm

[Roundup Robot]

Roundup Robot added the comment:


New changeset aa26a5712a80849e968171b71b6bc8d9da3ac163 by Stefan Krah in branch 
'master':
Issue29439: _decimal on Android requires linking with libm.
https://github.com/python/cpython/commit/aa26a5712a80849e968171b71b6bc8d9da3ac163


--

___
Python tracker 
<http://bugs.python.org/issue29439>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29439] _decimal on Android requires linking with libm

[Roundup Robot]

Roundup Robot added the comment:


New changeset aa26a5712a80849e968171b71b6bc8d9da3ac163 by Stefan Krah in branch 
'3.6':
Issue29439: _decimal on Android requires linking with libm.
https://github.com/python/cpython/commit/aa26a5712a80849e968171b71b6bc8d9da3ac163


--

___
Python tracker 
<http://bugs.python.org/issue29439>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29439] _decimal on Android requires linking with libm

[Roundup Robot]

Roundup Robot added the comment:

New changeset b60b46ad8751 by Stefan Krah in branch '3.6':
Issue29439: _decimal on Android requires linking with libm.
https://hg.python.org/cpython/rev/b60b46ad8751

--
nosy: +python-dev

___
Python tracker 
<http://bugs.python.org/issue29439>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29439] _decimal on Android requires linking with libm

[Stefan Krah]

Stefan Krah added the comment:

Thanks.  Strange that on other systems the compilers don't complain (usually 
they do).

--
assignee:  -> skrah

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29439] _decimal on Android requires linking with libm

[Chi Hsuan Yen]

New submission from Chi Hsuan Yen:

Just like issue21668, _decimal requires -lm on Android. This wasn't fixed 
because _decimal didn't build before issue26846 lands. More specificially, 
log10 is called 
https://hg.python.org/cpython/file/tip/Modules/_decimal/libmpdec/mpdecimal.c#l7862

Added _decimal and Android experts

--
components: Cross-Build
files: decimal.patch
keywords: patch
messages: 286906
nosy: Alex.Willmer, Chi Hsuan Yen, skrah, xdegaye
priority: normal
severity: normal
status: open
title: _decimal on Android requires linking with libm
type: behavior
versions: Python 3.7
Added file: http://bugs.python.org/file46506/decimal.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com