[issue27643] test_ctypes fails on AIX with xlc
aixtools added the comment:
On 30-Jul-16 02:51, Martin Panter wrote:
> Martin Panter added the comment:
>
> Looks like your Python 3 build is messed up. Maybe it doesn’t like running
> from a different directory. I would try from the main build directory, and
> note the test_bitfields has an S:
I would not know why - it is just a copy of the sources and then
configure, make, make DESTDIR=/some/where install (rather than make
install).
Same procedure I use for python2.7.
>
> ./python -m unittest -v ctypes.test.test_bitfields
No changes: 3.5.1
root@x064:[/data/prj/aixtools/python/python-3.5.1.2]./python -m unittest
-v ctypes.test.test_bitfields
test_anon_bitfields (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_c_wchar (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_longlong (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_mixed_1 (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_mixed_2 (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_mixed_3 (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_mixed_4 (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_multi_bitfields_size (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_nonint_types (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_signed (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_single_bitfield_size (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_uint32 (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_uint32_swap_big_endian (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_uint32_swap_little_endian (ctypes.test.test_bitfields.BitFieldTest)
... ok
test_uint64 (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_ulonglong (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_unsigned (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_ints (ctypes.test.test_bitfields.C_Test) ... FAIL
test_shorts (ctypes.test.test_bitfields.C_Test) ... FAIL
==
FAIL: test_ints (ctypes.test.test_bitfields.C_Test)
--
Traceback (most recent call last):
File
"/data/prj/aixtools/python/python-3.5.1.2/Lib/ctypes/test/test_bitfields.py",
line 41, in test_ints
self.assertEqual(getattr(b, name), func(byref(b),
name.encode('ascii')))
AssertionError: -1 != 1
==
FAIL: test_shorts (ctypes.test.test_bitfields.C_Test)
--
Traceback (most recent call last):
File
"/data/prj/aixtools/python/python-3.5.1.2/Lib/ctypes/test/test_bitfields.py",
line 48, in test_shorts
self.assertEqual(getattr(b, name), func(byref(b),
name.encode('ascii')))
AssertionError: -1 != 1
--
Ran 19 tests in 0.013s
FAILED (failures=2)
> What I am suggesting as a fix is to change line 381 from plain “int” to
> “signed int”, and 382 to “signed short”. I can make a patch later if that
> will help. Hopefully with these changes the C compiler will use signed
> integer logic, matching what I believe the ctypes library uses for c_int and
> c_short.
Working from Python-2.7.12 - added your changes, test still fails:
root@x064:[/data/prj/aixtools/python/python-2.7.12.0]./python -m
unittest -v ctypes.test.test_bitfields
test_anon_bitfields (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_c_wchar (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_longlong (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_mixed_1 (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_mixed_2 (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_mixed_3 (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_mixed_4 (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_multi_bitfields_size (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_nonint_types (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_signed (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_single_bitfield_size (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_uint32 (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_uint32_swap_big_endian (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_uint32_swap_little_endian (ctypes.test.test_bitfields.BitFieldTest)
... ok
test_uint64 (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_ulonglong (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_unsigned (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_ints (ctypes.test.test_bitfields.C_Test) ... ok
test_shorts (ctypes.test.test_bitfields.C_Test) ... FAIL
==
FAIL: test_shorts (ctypes.test.test_bitfields.C_Test)
--
Traceback (most recent call last):
File
"/data/prj/aixtools/python/python-2.7.12.0
[issue27435] ctypes and AIX - also for 2.7.X (and later)
New submission from aixtools:
I am opening a new issue # - about the same problem described in 26439, but now
calling it a bug
rather than a behavior change. Then I did not know better - all was new, and as
the "new" kid
I felt uncomfortable calling it a bug - when maybe it was just behavior I did
not understand
However, with the feedback from code review I am beginning to understand what
the ctypes functionality
is: ctypes.find_library() and cdll.LoadLibrary (aka CDLL)
in __init__.py it is clear that there is one entry point for LoadLibrary. For
find_library(),
I assume overtime, the development led to six (6) definitions - and I am
proposing a 7th.
The pseudo_code is roughly this (although import ctypes._$_platform._xxx may be
incorrect syntax)
def find_library(name)
import os,sys
_platform = sys.platform
import ctypes._$_platform._find_library as _find_library
return(_find_library(name))
Unclear is how a regular .so file should be treated by find_library() when the
actual file is not
prefixed by the string "lib", e.g., sudoers.so - should it only look for
libsudoers.so even though
Currently it is "defined" 7 times (including the definition I added at line 79)
because - again - util.py defines find_library several times - only one of the
definitions is called
dependent on the os.name or os.name + platform.
Within the os.name() and sys.platform code block there are sufficient
"liberities" to call
external programs used to determine where (i.e., find) the library is - and
what it's correct
prefix and suffix is.
+6 # find_library(name) returns the pathname of a library, or None.
+49 def find_library(name): in block if os.name == "nt"
+71 def find_library(name): if os.name == "ce" return None
+79 def find_library(name): if sys.platform.startswith("aix") return
ctypes._aixutils.find_library(name)
+84 def find_library(name): elif os.name == "posix" and sys.platform ==
"darwin":
+95 elif os.name == "posix":
+177 def find_library(name): inside: if
(sys.platform.startswith("freebsd")
or sys.platform.startswith("openbsd")
or sys.platform.startswith("dragonfly")):
+217 def find_library(name, is64 = False): inside:elif
sys.platform == "sunos5":
+220 else:
+249 def find_library(name):
+250 return _findSoname_ldconfig(name) or
_get_soname(_findLib_gcc(name))
++
Getting back to my patch development - initially I was adding code to both
util.py and __init__.py
and putting everything in " startswith("aix")" blocks.
As I thought, in error, find_library() was to make visible what CDLL was
finding on it's own
I was also doing find_library() like behavior. Rather than have the same code
in two files
I moved the code into a new file - now named _aixutils.py.
So, this looks like a design change, but it is not. All this code could be
re-indented and inserted
into util.py. IMHO, in a separate file the changes are easier to follow.
If a seperate file means it cannot be accepted into python-2.7.X then the code
needs to be moved back into
util.py
Note: the extra tests added to util.py are only for my enjoyment, i.e.,
verification of several cases I have
experienced while porting other packages. Think of them as not being there (for
now, if not for always).
Once I knew what CDLL(None) was suppossed to
That is why my first patch had CDLL() also calling ctypes._aixutils - to only
edit once.
Once I learned how wrong I was - that was removed - and it just remained this
way as "eye-candy".
+++ Closing +++
find_library() is defined multiple times, so adding one for aix is not a design
change,
i.e., behavior modification rather, find_library - as defined for aix in
"default block"
(elif os.name = "posix": ... else: ...) specifically calling
_findSoname_ldconfig(name) or _get_soname(_findLib_gcc(name)) is a "bug of
ommission"
because the programs the two routines called depend on: ldconfig and gcc are
not, normally
present on AIX.
My patch is a different way changing line 250 from
return _findSoname_ldconfig(name) or _get_soname(_findLib_gcc(name))
to
return _findSoname_ldconfig(name) or _get_soname(_findLib_gcc(name))
or _findAIXmember(name) or _find_Soname_dump(name)
with, of course def _findAIXmember() and def _findSoname_dump() defined within
the limits of util.py
Instead, I have "hard-coded" an approach similar to the psuedo code example.
+++ Disclaimer :P +++
As I am still relatively new, if I am still not understanding the official
definition of ctypes.find_library() - I apologize.
util.py is, due to is many defines of find_library() was difficult
for me to grasp the difference between permissible and "not done&
[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)
aixtools added the comment: On 6/12/2016 11:41 AM, Martin Panter wrote: > . > Michael, how are you supposed to apply your latest patch? I have Gnu Patch > 2.7.5, but even in the best case it doesn’t find the files to patch: > > $ patch -p1 -n < python.Lib.ctypes.160608.patch > can't find file to patch at input line 2 > The text leading up to this was: > -- > |diff -r Python-2.7.11/Lib/ctypes/__init__.py > python-2.7.11.5/Lib/ctypes/__init__.py > -- I must have forgotten the u (-r rather than -ru) (read mistyped, as I "always" use the -u option) > Perhaps can you make a “unified” diff (-u option) like you did in your > previous patches? Whatever you did with Python3.issue26439.160511.patch > seemed to work best. Will redo all of them in about 24 hours. > Also, since your patches add new functionality (the automatic archive member > detection), they will have to be for Python 3, not 2. Well, let me know what needs to be deleted - I see the member detection as the equivalent of reading the output of "ldconfig -p" to search for where an archive might be, or compareable with the gcc command (forgot the options) to say where it is looking. Not going into Python2 misses the whole point from my perspective. Unfortunate (for my perspective) is that noone else is saying anything one way or the other. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26439> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)
aixtools added the comment:
On 6/10/2016 6:33 AM, Martin Panter wrote:
> Martin Panter added the comment:
>
> Will try to change the existing code from os.popen to subprocess (Issue
> 26439) to set a better example for new code like this
>
> --
> dependencies: +avoid using a shell in ctypes.util: replace os.popen with
> subprocess
>
> ___
> Python tracker <rep...@bugs.python.org>
> <http://bugs.python.org/issue26439>
> ___
I have added comments to _aixutil.py - but while doing so I came up with
some thoughts re "special cases", where
a) the member is not in a archive (i.e., looking for a file)
b) the filename does not begin with "lib"
c) CDLL.LoadLibrary(name) succeeds
So, given that /usr/lib is default search path, and file FOO.so exists
as /usr/lib/FOO.so - should find_library(name) return name, or None.
d) name = ../some/where/Foo.so; CDLL.LoadLibrary("../some/where/Foo.so")
works - should find_library(name) return name
e) assume find_library("../some/where/Foo") returns
../some/where/libFoo.so, or None - again, given that
../some/where/libFoo.a does not exist
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue26439>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)
aixtools added the comment:
Answering this again - now that the new patch is ready.
On 04-Jun-16 16:24, Martin Panter wrote:
> Martin Panter added the comment:
>
> Okay here are some more thoughts about your latest patch:
>
> ## Automatic RTLD_MEMBER ##
>
> I was still uneasy about the automatic setting of RTLD_MEMBER. But I looked
> for how others handle this, and I found Libtool’s LTDL library, and Apache
> Portable Runtime (APR). Both have a similar (but stricter) automatic addition
> based on detecting the archive(member) notation:
>
> http://git.savannah.gnu.org/cgit/libtool.git/commit/libltdl/loaders/dlopen.c?id=8fa719e
> https://github.com/apache/apr/commit/c27f8d2
>
> So I guess I can accept this way, if you think it is better than my other
> ideas:
a) I think it is more in line with what I perceive as normal practice
libFOO = CDLL(find_library("FOO"))
As CDLL() has always had a 'simple' string as input and not a "tuple".
I have also added some lines to test/test_loading.py to test direct
calling of CDLL() with fixed strings and a test of os.maxsize
and in util.py - but using e.g., CDLL(find_library("c") as behavior is
dependent on 32 or 64-bit mode
depending on mode - different output:
note: find_library("libssl64") is expected to return None - as it
would be "abnormal" to have an archive libssl64.a or a file libssl64.so
cd Lib/ctypes
../../python util.py
# 32-bit mode:
None
libc.a(shr.o)
libbz2.a(libbz2.so)
find_library("c")returns: libc.a(shr.o)
find_library("libc") returns: libc.a(shr.o)
find_library("libssl") returns: libssl.a(libssl.so)
find_library("libssl64") returns: None
find_library("ssl") returns: libssl.a(libssl.so)
find_library("libiconv") returns: libiconv.a(libiconv.so.2)
find_library("intl") returns: libintl.a(libintl.so.8)
libcrypt.a(shr.o)
# 64-bit mode:
None
libc.a(shr_64.o)
libbz2.a(libbz2.so.1)
find_library("c")returns: libc.a(shr_64.o)
find_library("libc") returns: libc.a(shr_64.o)
find_library("libssl") returns: libssl.a(libssl64.so.1.0.0)
find_library("libssl64") returns: None
find_library("ssl") returns: libssl.a(libssl64.so.1.0.0)
find_library("libiconv") returns: libiconv.a(shr4_64.o)
find_library("intl") returns: libintl.a(libintl.so.1)
libcrypt.a(shr_64.o)
>
> # Always uses RTLD_MEMBER, never loads a plain file outside an archive
> name = "libcrypto.a(libcrypto.so.1.0.0)"
>
> # Other options, that could be returned by find_library() and would not
> conflict with a plain file name
> name = ("libcrypto.a", "libcrypto.so.1.0.0") # (archive, member)
> name = ("libcrypto.a(libcrypto.so.1.0.0)", RTLD_MEMBER) # (name, extra-flags)
>
> libcrypto = CDLL(name)
>
> ## find_library() modes ##
>
> In your find_library() function, you still have three parts. Can you confirm
> that each behaviour is intended:
I was being "Q" here, not changing the aixutils.py (now _aixutils.py).
My intent is to be comparable with other "posix" behaviors.
So, if you believe I am still not compatible with their behavior,
forgive me - but also shoot me!
>
> A) If I have a file called "crypto" in the current directory,
> find_library("crypto") returns "crypto". This does not seem right. On Linux,
> “gcc [. . .] -lcrypto” does not look for a file exactly called “crypto”.
>
> B) You are still stripping bits off the library name if it contains “lib” or
> a dot (.), so find_library("glib-2.0") is almost equivalent to
> find_library("b-2"). Isn’t this a bug?
>
> C) find_library("crypto") will return "/usr/lib/crypto" if such a file
> exists. Just like in A), this does not seem right to me.
All should be seen as bugs, and I hope I coded it correctly to not do
this anymore.
> ## Other things ##
>
> * You don’t need to prefix most names with underscores, unless they could be
> confused with a public API. If you follow my earlier suggestion of renaming
> the new file to _aixutil.py (so it is obvious it is not a public module),
> then you can freely write “import re, os, sys”, etc.
I had missed, certainly not understood the context, before. aixutil.py
is now _aixutil.py. Originally I had done this to make the diff in
util.py much simplier, but also because I incorrectly thought CDLL() was
frequently called with "foo" or "libfoo". In short, trying to prevent a
non-existent problem.
__init__.py delta is also much much simpler to grasp.
> * No need to add the internal variable names to the function signatures. Just
> write find_library(name), and i
[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)
aixtools added the comment:
On 04-Jun-16 16:24, Martin Panter wrote:
> Martin Panter added the comment:
>
> Okay here are some more thoughts about your latest patch:
>
> ## Automatic RTLD_MEMBER ##
>
> I was still uneasy about the automatic setting of RTLD_MEMBER. But I looked
> for how others handle this, and I found Libtool’s LTDL library, and Apache
> Portable Runtime (APR). Both have a similar (but stricter) automatic addition
> based on detecting the archive(member) notation:
>
> http://git.savannah.gnu.org/cgit/libtool.git/commit/libltdl/loaders/dlopen.c?id=8fa719e
> https://github.com/apache/apr/commit/c27f8d2
The "stricter" from libtool is an excellent idea - I should done that as
well, but I really hate string manipulation :)
I hope that the intent at least more clear, and should anyone ever
complain about it not opening a file named: libFoo.a(libFOO.so) as a
filename - by design, not accepted as a filename.
> So I guess I can accept this way, if you think it is better than my other
> ideas:
>
> # Always uses RTLD_MEMBER, never loads a plain file outside an archive
> name = "libcrypto.a(libcrypto.so.1.0.0)"
>
> # Other options, that could be returned by find_library() and would not
> conflict with a plain file name
> name = ("libcrypto.a", "libcrypto.so.1.0.0") # (archive, member)
> name = ("libcrypto.a(libcrypto.so.1.0.0)", RTLD_MEMBER) # (name, extra-flags)
>
> libcrypto = CDLL(name)
Isn't this more of an API change - name is no longer just a string?
>
> ## find_library() modes ##
>
> In your find_library() function, you still have three parts. Can you confirm
> that each behaviour is intended:
I have to catch a plane - will get back on these. Short - if I have a
potential bug, then needs to be improved. More later.
>
> A) If I have a file called "crypto" in the current directory,
> find_library("crypto") returns "crypto". This does not seem right. On Linux,
> “gcc [. . .] -lcrypto” does not look for a file exactly called “crypto”.
>
> B) You are still stripping bits off the library name if it contains “lib” or
> a dot (.), so find_library("glib-2.0") is almost equivalent to
> find_library("b-2"). Isn’t this a bug?
>
> C) find_library("crypto") will return "/usr/lib/crypto" if such a file
> exists. Just like in A), this does not seem right to me.
>
> ## Other things ##
>
> * You don’t need to prefix most names with underscores, unless they could be
> confused with a public API. If you follow my earlier suggestion of renaming
> the new file to _aixutil.py (so it is obvious it is not a public module),
> then you can freely write “import re, os, sys”, etc.
>
> * No need to add the internal variable names to the function signatures. Just
> write find_library(name), and if you need to initialize a variable, do that
> in the body.
>
> * I suggest to go over all the regular expressions, and either change them to
> plain string searching, or make sure special characters and external
> variables are escaped as necessary. A comment explaining what the RE is
> trying to do might help too.
>
> --
>
> ___
> Python tracker <rep...@bugs.python.org>
> <http://bugs.python.org/issue26439>
> ___
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue26439>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)
aixtools added the comment:
On 28-May-16 04:04, Martin Panter wrote:
> Martin Panter added the comment:
>
> Strictly speaking, Python 2.7 never had special support for RTLD_MEMBER or
> find_library(). That is why I am unsure about many of these changes being
> done in 2.7. They seem more like new features than bug fixes. Especially once
> you start talking about a smart CDLL() that tests for a file and then falls
> back to RTLD_MEMBER.
Well, actually, within AIX I was intending to "just use" RTLD_MEMBER
when "the string" ended in a ")" as that is the correct additional mode
bit for an established interface dlopen(). I perhaps do not understand
(certainly did not) the goal of CDLL(), better Lib/ctypes
As far as Python-2.7 is concerned - I would like to see it "in" because
there is a lot of Python-2(.7) code out there that will not work without
manageling AIX at a low level (i.e., current practice is to extract .so
members from archives so dlopen() will be able to open them.
>
> Whether you represent an archive member as an (archive, member) tuple, or as
> an "archive(member)" string, or something else, it is still an extension of
> the API of CDLL(), and arguably also of find_library().
I would want to disagree. A string is a string, regardless of the
characters in it. And, I have verified - if a file
/usr/lib/libFOO.a(libFoo.so.X) exists dlopen() will open it when
RTLD_MEMBER is not ORed into the mode.
> I think I have already tried to explain that on Linux,
> find_library("libm.so") is far from normal usage (although may be technically
> legal if there was a library called liblibm.so.so).
find_library("libm.so") is far from normal usage: again, my
misunderstanding - I was reading as much as possible from various
sources - and my head was spinning.
In util.py there was no "libFOO.so", only "FOO" and "libFOO" - so my
find_library() tests are not correct. Agreed.
>
> Also, on Linux I wouldn’t recommend CDLL("libcrypt.so"), which relies on a
> file only intended for build time. On Debian, the glibc library
> <https://packages.debian.org/source/jessie/glibc> has separate packages for
> shared libraries (libc6) and development libraries (libc6-dev). It should be
> possible to run programs with libc6 installed, but without libc6-dev. Looking
> at the file lists for “i386” architecture, libc6-dev includes:
>
> /usr/lib/i386-linux-gnu/libcrypt.a
> /usr/lib/i386-linux-gnu/libcrypt.so
My expectation (misconception perhaps) is that Linux uses a .a archive
for static linking, and .so shared-library for dynamic-loading. The AIX
loader/linker looks first in .a archives for shared libraries (when
static is not specified) and if no member is found (member name is not
relevant for the search, only the symbol and the object size (32 or 64
bit)). If not found in libFOO.a then a search for a specific file is
done libFOO.so.
An assumption of mine is: if I specified (not usual on AIX, based on my
experience) -lc.so.6 AIX would look for an archive named libc.so.6.a
first. If found, would look for a member in that archive. If not found
as a .a archive and/or not found as a file at all then it will look for
a file libFOO.so.6
>
> It symbolically links to /lib/i386-linux-gnu/libcrypt.so.1, which is provided
> by libc6:
>
> /lib/i386-linux-gnu/libcrypt-2.19.so
> /lib/i386-linux-gnu/libcrypt.so.1
>
> So on Debian (and with Linux in general), CDLL("libcrypt.so") may work in
> some cases, but you should normally use CDLL("libcrypt.so.1") instead.
Agreed - CDLL("libFOO.so") is not preferred. But, should it be refused?
Or, if it is accepted on Linux - and works when the (symbolic) file
exists - must it be refused on AIX? If refusing it on AIX is what it
takes to make the patch acceptable - then code it to refuse it is.
>
> Usually people just upload a new version of the original patch (original
> changes, plus new changes, in one file). I think it would be too confusing
> looking at differences between two patches.
So, just provide the new delta between "2.7.11(.0) and what I have, not
the changes between patches. Coming up tomorrow.
In short, my head was spinning and I thought I saw problems that are not
meant to exist, in particular find_library("libFOO.so") - that should be
coded to reject I guess. I would still like to let something like
"libc.so" be creative, as libc.so will never exist, by default, on AIX.
My guess is that is a legacy thing from AIX4. (and I think the default
name is shr.o when using the ld flag -G to create a shared object aka
shared library)
>
> --
>
> ___
> Python tracker <rep...@bugs.python.org>
> <http://bu
[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)
aixtools added the comment:
See below between points.
On 12-May-16 17:45, Martin Panter wrote:
> Martin Panter added the comment:
>
> Thanks for trying to move the directory in the new patch, but it does not
> seem to have been picked up by Rietveld. I have only seen patches generated
> with Mercurial and Git work, but I don’t exactly know _why_ they work :)
>
> Here is a summary of what I would be comfortable adding to Python:
>
> 1. Add ctypes.RTLD_MEMBER constant (new API, so 3.6+ only)
Understood. Should also add RTLD_NOW as it is referenced in comments,
and used in the _global.c (imho).
> 2. Basic ctypes.util.find_library("crypto") implementation for AIX. But cases
> like find_library("libintl.so") should fail. I think your earlier patches
> were closer to this implementation than the later ones.
>
> I am a bit hesitant about the automatic behaviour of
> CDLL("libcrypto.a(libcrypto.so.1.0.0)") using RTLD_MEMBER. IMO it may be
> better to let the caller specify RTLD_MEMBER explicitly. If a shared library
> file literally called “/usr/lib/libcrypto.a(libcrypto.so.1.0.0)” existed,
> i.e. not inside an archive, would dlopen("libcrypto.a(libcrypto.so.1.0.0)",
> RTLD_NOW) succeed? I admit this is an unlikely scenario, but it seems bad to
> reduce the domain of a low-level API.
>
> I understand it would be good to have the return value of find_library()
> consistent with the name accepted by CDLL(). Perhaps a new parameter format
> would help, such as a tuple (archive, member).
I would expect a new tuple would be seen as new API (like
_ctypes.RTLD_MEMBER), so it would never help in Python-2.7, which is
still where I see most people asking for support with AIX.
My guidance was: find_library("m") is legal, as is
find_library("libm.so"), as is CDLL("libcrypt.so").
These are all dependencies, at a low-level of how the system is
configured, i.e., it is not uncommon for a GNU/Linux installation to
have many symbolic links to resolve multiple names, and programmers use
one, or more of those conventions. The find_library() and CDLL() calls
work because of these conventions.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Thu May 19 02:53:48 2016 from xxx.yyy.zzz.net
<http://76-217-205-93.lightspeed.sndgca.sbcglobal.net>
root@x066:~# python -m ctypes.util
libm.so.6
libc.so.6
libbz2.so.1.0
libcrypt.so.1
Looking at filesystem level:
root@x066:~# cd /lib64
root@x066:/lib64# ls -l libcrypt.*
lrwxrwxrwx 1 root root 16 Feb 22 2015 libcrypt.so.1 -> libcrypt-2.13.so
<http://libcrypt-2.13.so>
root@x066:/lib64# cd /usr/lib64
root@x066:/usr/lib64# ls -l libcrypt.*
-rw-r--r-- 1 root root 62108 Feb 22 2015 libcrypt.a
lrwxrwxrwx 1 root root20 Feb 22 2015 libcrypt.so ->
/lib64/libcrypt.so.1
So, CDLL.dlopen() finds /usr/lib64/libcrypt.so that is a symbolic link
to /lib64/libcrypt.so.1 that is a symbolic link to
/lib64/libcrypt-2.13.so <http://libcrypt-2.13.so>. I see libcrypt.so as
a "shorthand" for whatever is installed. As a porgrammer/user of
CDLL.dlopen() I do not care what it finally resolves to.
Linux has it's conventions: one important one I observe when porting OSS
packages is that "installation" and updates of libraries (or API's)
provide "shortnames" in the form of libFOO.so that are symbolic names
to the final object. Unfortunately, there are exceptions - especially
look at "libc.so"
root@x066:/usr/lib64# ls -l libc.so*
-rw-r--r-- 1 root root 243 Feb 22 2015 libc.so
root@x066:/usr/lib64# cd /lib64
root@x066:/lib64# ls -l libc.so*
lrwxrwxrwx 1 root root 12 Feb 22 2015 libc.so.6 -> libc-2.13.so
root@x066:/lib64# ls -l libc-2.13.so
-rwxr-xr-x 1 root root 1758016 Feb 22 2015 libc-2.13.so
Obviously, using shorthand for libc.so on Debian is not returning
anything equivalent to libc.so.6 (look at the different sizes). So, a
program that looks specifically for libc.so.6 will succeed on Debian,
and fail on AIX (as it does not exist in that form) - and my expectation
is that existing programs that code libc.so.6 will fail on AIX. However,
I have extreme doubts that anyone actually codes the test - as (at least
on AIX) libc is already loaded.
AIX:
root@x071:[/root]ldd /opt/bin/python
/opt/bin/python needs:
/usr/lib/libc.a(shr.o)
/usr/lib/libpthreads.a(shr_xpg5.o)
/usr/lib/libpthreads.a(shr_comm.o)
/usr/lib/libdl.a(shr.o)
/unix
/usr/lib/libcrypt.a(shr.o)
And, it seems, libc is also already loaded on Debian:
root@x066:/lib64# ldd /usr/bin/python
linux-vdso32.so.1 => (0x0010)
libpthread.so.0 => /lib/powerpc-linux-gnu/libpthread.so.0
(0x0ffc5000)
libdl.so.2 => /lib/powerpc-linux-gnu/libdl.so.2 (0x
[issue11063] uuid.py module import has heavy side effects
aixtools added the comment: The way I have seen that resolved - in many locations, is to have the option to specify a specific version, e.g., libFOO.so.1 and then as long as that version remains available, perhaps as read-only for running with existing programs,they continue to work even when libFOO.so.2. However, for some who believes, or expects, to be able to deal with potential ABI changes - they can specify simply libFOO.so and let the loader decide (what I see is that libFOO.so is a symbolic link to, or a copy of the latest version.) So, I agree wholeheartedly, if a versioned number is requested, that, or nothing, should be returned. However, if it is generic - try to find a generic named library (and get the link or copy), and when that is not available either, take the latest versioned number. It has been over two months, and I may have read it wrong - but that appears to be what the current "ldconfig -p" solution implements. (in ctypes, not uuid, so perhaps this is not the correct place to be responding. If so, my apologies). On 08-May-16 05:24, Martin Panter wrote: > Martin Panter added the comment: > > The versioning problem with libFOO.so.N already occurs with compiled > programs. A C program compiled against libuuid.so.1 will fail to load if you > only have libuuid.so.2. On the other hand, a Python program using > find_library() will find either version. My point about robustness is that if > a version 2 is invented, it might have different semantics or function > signatures, and Python would then be assuming the wrong semantics. > > -- > > ___ > Python tracker <rep...@bugs.python.org> > <http://bugs.python.org/issue11063> > ___ -- nosy: +aixto...@gmail.com ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue11063> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)
aixtools added the comment:
On 5/8/2016 8:29 AM, Martin Panter wrote:
> Martin Panter added the comment:
>
> Your new patch calls find_library() internally in CDLL(); why?
Because arguments that work for GNU (aka Linux, even though internally
it is called "posix") will not work for AIX.
> My understanding is CDLL() is a fairly lightweight wrapper around the
> dlopen() call.
For AIX, the dlopen() call is a front-end for the function
initAndLoad(). I have not looked in depth at Module/_ctypes (only
recently found it) and I do not know how much work is being done there
for the different platforms.
> On Linux, you either pass a full library file name, or an SO-name.
Same for AIX: a full-path library name, relative-path library name, or a
library name and the library search path (with optional additional
directories provided via environment variables (LIBPATH or
LD_LIBRARY_PATH - when it is a FILE, or an archive(member_name) (with
archive part being full or relative, or stem) and RTLD_MEMBER ORed into
the mode argument.
> Both these strings can be discovered for compiled objects using e.g.:
Likewise for AIX: - example: an executable - notice how many are named
shr.o: how is cdll("shr.o") suppossed to know it is libc, or is it libcrypt?
root@x064:[/usr/lib]ldd /usr/sbin/sshd
/usr/sbin/sshd needs:
/usr/lib/libc.a(shr.o)
/usr/lib/libcrypto.a(libcrypto.so.1.0.0)
/usr/lib/libz.a(libz.so.1)
/usr/lib/libpam.a(shr.o)
/usr/lib/libdl.a(shr.o)
/unix
/usr/lib/libcrypt.a(shr.o)
/usr/lib/libpthreads.a(shr_xpg5.o)
/usr/lib/libpthreads.a(shr_comm.o)
Or a file:
./perl-5.14.4/lib/5.14.4/aix-thread-multi-64int/auto/B/B.so needs:
/usr/lib/libpthreads.a(shr_xpg5.o)
/usr/lib/libc.a(shr.o)
/unix
/usr/lib/libpthreads.a(shr_comm.o)
/usr/lib/libcrypt.a(shr.o)
Again, cdll("shr.o") is not going to find the correct archive.
> $ ldd build/lib.linux-x86_64-2.7-pydebug/_ssl.so
> linux-vdso.so.1 (0x7fff567fe000)
> libssl.so.1.0.0 => /usr/lib/libssl.so.1.0.0 (0x7f598474c000)
> libcrypto.so.1.0.0 => /usr/lib/libcrypto.so.1.0.0 (0x7f59842d4000)
> . . .
>
> So in Python, the SO-name or full path can be used, but not the compile-time
> name, unless you first pass it through find_library():
>>>> CDLL("libcrypto.so.1.0.0") # soname
>
>>>> CDLL("/usr/lib/libcrypto.so.1.0.0") # Full path
>
>>>> CDLL("crypto") # Compile-time name
> Traceback (most recent call last):
>File "", line 1, in
>File "/usr/lib/python2.7/ctypes/__init__.py", line 365, in __init__
> self._handle = _dlopen(self._name, mode)
> OSError: crypto: cannot open shared object file: No such file or directory
>>>> find_library("crypto") # Some people pass the result of this to CDLL()
> 'libcrypto.so.1.0.0'
Thank you for the explanation of what should be used when. Maybe I
worked too hard for the current result: (note - 64-bit this time! and
new member names, even for libc.so)
michael@x071:[/usr/lib]python -m ctypes.util
None
libc.a(shr_64.o)
None
aix.find_library("libiconv.so") libiconv.a(shr4_64.o)
aix.find_library("libintl.so") libintl.a(libintl.so.1)
aix.find_library("libintl.so.1") libintl.a(libintl.so.1)
aix.find_library("libintl.so.2") None :: should be None!
aix.find_library("libintl.so.8") None
libcrypt.a(shr_64.o)
Back to Why though.
Ease of use, and portability with existing convention. My experience
with just two python packages I wanted to port is that cdll("libFOO.so")
is what is used, and works on GNU. I wanted that to work ASIS. I also
noticed that sometimes code was explicit about the version (sometimes
the latest was not right, and the code complained). So, since "short"
names are preferred, and are portable - those are what programmers tend
to use. Full-path-names imply an assumption.
dlopen() is not going to know that libc.so is actually libc.a(shr.o) -
so I call find_library() for portability aka Ease of Use. While it may
be slightly less performance I am hoping (assuming) cdll("foo") is not
the most central part of any application - and the overhead is acceptable.
And thanks for the question. It is simple enough to remove now, should
it be considered a sin.
> --
>
> ___
> Python tracker <rep...@bugs.python.org>
> <http://bugs.python.org/issue26439>
> ___
--
nosy: +aixto...@gmail.com
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue26439>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6006] ffi.c compile failures on AIX 5.3 with xlc
aixtools added the comment: Although this is closed - it is not (yet) resolved, really. ctypes does not compile on AIX 5.3, or AIX 6.1. Applying the patch posted (https://bugs.python.org/file17098/pyffi.patch) does get rid of most of the problems. However, to completely fix it I also copied as follows: cp -rp ../sourceforge/libffi/libffi-3.2.1/src/powerpc ./Python-2.7.8/Modules/_ctypes/libffi/src/powerpc make clean make There are still other issues - BUT - with this ctypes completes as a module of python. -- nosy: +aixto...@gmail.com versions: +Python 2.7 -Python 3.0 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6006 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24046] Incomplete build on AIX
New submission from aixtools:
Actually, I have been building and using my builds of Python, when needed for
./configure requirements for a long time. In short, it is quite nice that make
completes even when there are missing and/or failed modules.
I have just resolved the problem with ctypes not building (see
https://bugs.python.org/issue6006) and that got me started to research others.
Failed to build these modules:
_elementtree _sqlite3 _ssl
bz2pyexpat
While there are several - I am looking first at ssl.
My first attempt comes up with some failed defines - probably because the
latest openssl provided by IBM is openssl-1.0.0 and openssl-1.0.1 is needed.
Rather than wait for that to happen I decided to experiment with LibreSSL. If
you are not familiar with LibreSSL - I shall be quick - openbsd (who also
maintains openssh) has been cutting out insecure and/or superfluous code.
One of the more insecure (because it can be a predictable source of enthropy)
is RAND_egd() - so it is unavoidable that this occurs:
ld: 0711-317 ERROR: Undefined symbol: .RAND_egd
After patching _ssl.c to this:
--- _ssl.c.orig 2014-06-30 02:05:42 +
+++ _ssl.c 2015-04-24 02:47:00 +
@@ -1604,6 +1604,7 @@
static PyObject *
PySSL_RAND_egd(PyObject *self, PyObject *arg)
{
+#ifndef LIBRESSL_VERSION_NUMBER
int bytes;
if (!PyString_Check(arg))
@@ -1618,6 +1619,12 @@
return NULL;
}
return PyInt_FromLong(bytes);
+#else
+PyErr_SetString(PySSLErrorObject,
+external EGD connection not allowed when using
LibreSSL:
+no data to seed the PRNG via PySSL_RAND_egd);
+return NULL;
+#endif
}
PyDoc_STRVAR(PySSL_RAND_egd_doc,
The end result is:
Failed to build these modules:
_elementtree _sqlite3 bz2
pyexpat
In short, you can get ahead of the curve by depreciating/removing
PySSL_RAND_egd() because any code that uses it may be receiving predictable
input and thereafter everything may be predictable.
If you do not believe openbsd (or me) - just read the code. It calls anything
configured (handy when /dev/urandom was hard to find anno 1999) but these days
a backdoor waiting to be opened.
p.s. As I get time I shall continue with the other modules that do not build -
just let me know if you prefer that I continue posting in this issue, or make
new one(s) for each module as I find a solution.
--
components: Extension Modules
messages: 241908
nosy: aixto...@gmail.com
priority: normal
severity: normal
status: open
title: Incomplete build on AIX
type: compile error
versions: Python 2.7
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24046
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
