Re: Finding .so files without setting LD_LIBRARY_PATH
On Thu, 2016-05-12 at 07:55 +0300, Jussi Piitulainen wrote: > eryk sun writes: > > > On Wed, May 11, 2016 at 10:39 PM, Paul Smith wrote: > > > Hi all. I have a locally-built version of Python (2.7.11) that I'm > > > copying around to different systems, running all different versions of > > > GNU/Linux. > > ... > > > What I'd like to do is have a way of setting the library path that > > > Python uses when it tries to load .so files (for example in the ssl > > > module which loads lib-dynload/_ssl.so which links to libssl.so and > > > libcrypto.so), WITHOUT setting LD_LIBRARY_PATH in the environment that > > > Python passes to its children. > > > > An ELF header can contain either an RPATH or a RUNPATH to extend the > > library search path at load time. Yes, I'm familiar with rpath. However I don't know how to set it for Python .so's that appear in the lib-dynload directory, including site -packages, which is where it needs to be ... I tried to go through the Modules/Setup etc. but to no avail :(. > There's a tool (GPLv3+) called patchelf that can set these in an ELF > binary: Aha! That's excellent! The only one I was aware of was chrpath which doesn't allow you to add an rpath if one doesn't exist, or add an rpath which is longer than the one that already exists. This will make things much simpler. Cheers! -- https://mail.python.org/mailman/listinfo/python-list
Finding .so files without setting LD_LIBRARY_PATH
Hi all. I have a locally-built version of Python (2.7.11) that I'm copying around to different systems, running all different versions of GNU/Linux. Because I need this to work across systems I'm bundling important .so's with my Python installation (libcrypto, libssl, libreadline, libgmp) which are dynamically loaded when I do things like "import ssl" etc. I have a wrapper script around this Python that everyone runs, rather than invoking it directly, and it is able to set some environment variables etc. So, I had been setting LD_LIBRARY_PATH before I invoke Python so it can find the "right" versions of libcrypto.so etc. That works fine, but here's the problem: because LD_LIBRARY_PATH is in Python's environment it is also passed down to programs invoked by Python. That means if I (for example) invoke subprocess.call(['ssh', ...]) then it fails because the system ssh is looking for the system libcrypto.so, and when it finds the Python libcrypto.so instead (because of LD_LIBRARY_PATH) it fails. What I'd like to do is have a way of setting the library path that Python uses when it tries to load .so files (for example in the ssl module which loads lib-dynload/_ssl.so which links to libssl.so and libcrypto.so), WITHOUT setting LD_LIBRARY_PATH in the environment that Python passes to its children. Is there any way to do this? It seems like there must be some way to do it, because if after I start python with LD_LIBRARY_PATH set, then I delete it from os.environ, then I import ssl it works, so it's still set in Python's environment. But I don't want to have to modify all my scripts to unset LD_LIBRARY_PATH, plus that's not the right thing to do in case someone needs to set LD_LIBRARY_PATH for themselves. What I've tried so far is writing a little Python script that unsets LD_LIBRARY_PATH and setting PYTHONSETUP to the name of that script in my wrapper, which does seem to work, but that takes away the ability for users to use PYTHONSETUP themselves which is sub-optimal. Does anyone have any other ideas? -- https://mail.python.org/mailman/listinfo/python-list
Re: Experiences/guidance on teaching Python as a first programming language
On Wed, 2013-12-18 at 01:33 +, Steven D'Aprano wrote:
> And "What does 'implementation-specific undefined behaviour' actually
> mean in practice?", another common question when dealing with C.
Only asked by people who haven't had it explained. There's "undefined
behavior", and there's "implementation-specific behavior", but it is
impossible to have "implementation-specific undefined behavior".
And, the definitions are simple to understand: "undefined behavior"
means that if your program invokes it, there is no definition of what
will happen. This is buggy code.
"Implementation-specific" behavior means that the standard requires the
implementation to do some well-defined thing, but the standard does not
define exactly what it must be. You can go look up what your
implementation will do in its documentation (the standard requires that
it be documented), but you can't assume the same thing will happen in
another implementation. This is non-portable code.
It's a very rare language indeed that has no undefined or
implementation-specific behaviors. Python gets to "cheat" by having one
reference implementation. Every time you've had to go try something out
in the Python interpreter because the documentation didn't provide the
details you needed, that WAS implementation-specific behavior.
> > You never have to wonder what the
> > lifetime of an object is,
>
> Since C isn't object oriented, the lifetime of objects in C is, um, any
> number you like. "The lifetime of objects in objects> is ONE MILLION YEARS!!!" is as good as any other vacuously true
> statement.
The implication that only an "object oriented" language could have a
concept of object lifetimes is false. Another, less hyperbolic way of
saying this is that in C, the lifetime of objects is _exactly as long as
you specify_. Heap objects come into existence when you explicitly
create them, and they go out of existence when you explicitly destroy
them. If you don't destroy them, they never go away. If you destroy
them more than once, that's undefined behavior. Stack objects are even
simpler.
> > or be mystified by which of the 7 signatures
> > of Foo.foo() are going to get called,
>
> Is that even possible in C? If Foo is a struct, and Foo.foo a member, I
> don't think C has first-class functions and so Foo.foo can't be callable.
Of course that's valid C. It's true that C doesn't have first-class
functions, but it supports invoking functions through pointers and you
can store functions in data members, pass functions as arguments, and
return functions from other functions, so Foo.foo can certainly be
callable.
~$ cat /tmp/foo.c
#include
struct Foo {
void (*foo)();
};
void foobar(void) { printf("foobar\n"); }
int main()
{
struct Foo Foo = { foobar };
Foo.foo();
return 0;
}
$ gcc -Wall -o /tmp/foo /tmp/foo.c
$ /tmp/foo
foobar
--
https://mail.python.org/mailman/listinfo/python-list
Re: Logger module in python
On Wed, 2013-12-18 at 03:27 +, Mark Lawrence wrote: > On 18/12/2013 03:22, smilesonisa...@gmail.com wrote: > > Hi, > > I am a newbie in python. I am looking for a existing module which I > can import in my program to log the objects to a file? > > > > I know there is a module Data::Dumper in perl which dumps the > objects to file. But not sure about python. > > http://docs.python.org/3/library/logging.html That's not what the OP wants. Pradeep, if you want to dump data for debugging take a look at the pprint module; for example: http://docs.python.org/3.3/library/pprint.html If you want to store Python objects in files and read them later, most people seem to use pickle for that: http://docs.python.org/3.3/library/pickle.html Others translate into another format. See this SO question for some other suggestions: http://stackoverflow.com/questions/2540567/is-there-a-python-equivalent-to-perls-datadumper -- https://mail.python.org/mailman/listinfo/python-list
Re: Sharing Python installation between architectures
On Sun, 2013-11-17 at 10:46 -0500, Paul Smith wrote: > Unfortunately, if you set PYTHONHOME then it's used for both $PREFIX and > $EXECPREFIX without any path probing whatsoever, so PYTHONHOME is > unusable with an installation where you've used different values for > --prefix and --exec-prefix during configure. Gak. Never mind. That's what you get when you're trying to hack relocatable installations at 3am: even after some sleep you think you know what's going on. PYTHONHOME accepts a colon-separated list for prefix:exec-prefix. This is even clearly documented, even in the error output from the interpreter when it can't find its installation! Making this change in my wrapper script for GDB gets everything working. Cheers! -- https://mail.python.org/mailman/listinfo/python-list
Re: Sharing Python installation between architectures
On Sat, 2013-11-16 at 19:28 -0500, Paul Smith wrote: > On Fri, 2013-11-15 at 18:00 -0500, Paul Smith wrote: > > By this I mean, basically, multiple architectures (Linux, Solaris, > > MacOSX, even Windows) sharing the same $prefix/lib/python2.7 directory. > > The large majority of the contents there are completely portable across > > architectures (aren't they?) so why should I have to duplicate many > > megabytes worth of files? > > OK, after some investigation and reading the code in Modules/getpath.c > to determine exactly how sys.prefix and sys.exec_prefix are computed (it > would be nice if this algorithm was documented somewhere... maybe it is > but I couldn't find it) I have a solution for this that appears to be > working fairly well. Ouch. I spoke a bit too soon. The standard Python installation and interpreter works fine with the "split --prefix and --exec-prefix" configuration, even with relocation of the installation directory. However, that configuration doesn't work for embedded Python (for example, if you embed the Python interpreter in GDB by linking libpython2.7.a) if you relocate it. In that configuration when the interpreter looks for the lib/python directory at runtime it appears to use only the prefix path Python was originally compiled with (I'm using strace on Linux to see what paths are being probed at startup). It doesn't use the "current installation path" via argv[0], which means it's not relocatable at all. I can, of course, set PYTHONHOME to point to the right place. Unfortunately, if you set PYTHONHOME then it's used for both $PREFIX and $EXECPREFIX without any path probing whatsoever, so PYTHONHOME is unusable with an installation where you've used different values for --prefix and --exec-prefix during configure. We'd need a new environment variable, like PYTHONEXECHOME or something, which would be tested first when looking for the exec_path (only). If that didn't exist, it could try PYTHONHOME as before. I'm willing to do this and file a bug with a patch if there's any interest in pursuing it further. Or should this be discussed on python-dev? -- https://mail.python.org/mailman/listinfo/python-list
grammar (was Re: Automation)
On Sat, 2013-11-16 at 10:11 -0500, Roy Smith wrote: > In article , > William Ray Wing wrote: > > > And my personal peeve - using it's (contraction) when its (possessive) > > should have been used; occasionally vice-versa. > And one of mine is when people write, "Here, here!" to signify > agreement. What they really mean to write is, "Hear, hear!", meaning, > "Listen to what that person said". The one that really irks me is people using "loose" when they mean "lose". These words are not related, and they don't sound the same. Plus this mistake is very common; I typically see it at least once a day. -- https://mail.python.org/mailman/listinfo/python-list
Re: Sharing Python installation between architectures
On Fri, 2013-11-15 at 18:00 -0500, Paul Smith wrote: > By this I mean, basically, multiple architectures (Linux, Solaris, > MacOSX, even Windows) sharing the same $prefix/lib/python2.7 directory. > The large majority of the contents there are completely portable across > architectures (aren't they?) so why should I have to duplicate many > megabytes worth of files? OK, after some investigation and reading the code in Modules/getpath.c to determine exactly how sys.prefix and sys.exec_prefix are computed (it would be nice if this algorithm was documented somewhere... maybe it is but I couldn't find it) I have a solution for this that appears to be working fairly well. In order to get this to work you need to use the following arguments when you run "configure" to build your Python: configure --prefix=$PREFIX --exec-prefix=$PREFIX/$ARCH where $ARCH can be pretty much whatever you want, but should be unique for each different architecture of course. The $PREFIX should be the same for all architectures. The magic here is ensuring that the --exec-prefix directory is a SUBDIRECTORY of --prefix. If that's not true, nothing works! The resulting python interpreter will live in $PREFIX/$ARCH/bin: you have to leave it there! If you move it nothing works. Although you can get rid of the "bin" and move it up into $PREFIX/$ARCH if you want; that's OK. What I do is have a little shell-script wrapper installed somewhere else that runs 'exec $EXECPREFIX/bin/python "$@"' You can also correctly install extra packages with setup.py, even if those packages have their own shared objects (like pycrypto or whatever). I should say, I've not thought about Windows yet. I don't know if this will work out for Windows. However, Windows is such a different beast anyway I think (at least in my environment) it will be OK to treat it separately and require Windows people to download/install their own Python. There are few nitty things that I needed to handle: 1. The _sysconfigdata.py file is put into $PREFIX not $EXECPREFIX, which is wrong since that file is very much architecture-specific. As a post-processing step I moved it from $PREFIX/... into $EXECPREFIX/.../lib-dynload. It's not quite correct since it's not a dynamic object, but lib-dynload is the only standard path on sys.paths from $EXECPREFIX. It works OK anyway. 2. All the scripts, even the ones in $PREFIX/bin, have hardcoded #! paths which go to a specific python in $EXECPREFIX/bin which is wrong (they can't be shared that way). I use a simple "sed -i" to replace them all with "#!/usr/bin/env python" instead. 3. There are some scripts that get dumped into $EXECPREFIX/bin rather than into $PREFIX/bin: "2to3", "idle", "pydoc", "smtpd.py". I think this is simply a bug in the installation and those should all go into $PREFIX/bin. Another weird thing is that the normal installation (this has nothing to do with the above; it happens even if you don't set --exec-prefix) contains TWO copies of libpython2.7.a; one in $EXECPREFIX/lib and one in $EXECPREFIX/lib/python2.7/config. These are over 14M each so it's not inconsequential to have two. I'm deleting the one in lib/python2.7/config and things still seem to work OK. The pkgconfig python definition references the one in $EXECPREFIX/lib. -- https://mail.python.org/mailman/listinfo/python-list
Sharing Python installation between architectures
One thing I always liked about Perl was the way you can create a single installation directory which can be shared between archictures. Say what you will about the language: the Porters have an enormous amount of experience and expertise producing portable and flexible interpreter installations. By this I mean, basically, multiple architectures (Linux, Solaris, MacOSX, even Windows) sharing the same $prefix/lib/python2.7 directory. The large majority of the contents there are completely portable across architectures (aren't they?) so why should I have to duplicate many megabytes worth of files? The only parts of the install which are not shareable (as far as I can tell) are the .so dynamic objects (and the python executable itself obviously). If the default sys.path included platform-specific directories as well as the generic lib-dynload, it would be possible. I do see that there are "plat-*" directories available in the default path. Is it possible to make use of these (say, by renaming each architecture's lib-dynload to the appropriate plat-* name)? If that works, the remaining issue is the site-packages directory. There is no ability (that I can see) to separate out the shareable vs. non-sharable aspects of the add-on site-packages. Any comments or suggestions? Am I overestimating the amount of sharing that's possible? Thanks! -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.x on MacOSX: failed dlopen() on .so's
On Thu, 2013-11-14 at 10:36 -0800, Ned Deily wrote: > In article <1384442536.3496.532.camel@pdsdesk>, > Paul Smith wrote: > [...] > > By relocatable I mean "runnable from any location"; i.e., not fixed. I > > have a wrapper around the Python executable that can compute the correct > > root directory and set any environment variables or add flags or > > whatever might be needed. > > In that case, the python.org installer may not be a good choice. You should > be to accomplish what you want by building your own Python. You'll probably > find you were getting tripped up by unnecessarily setting environment > variables. Good luck! Thanks Ned. I got sidetracked for a while but I got back to this now, and I found my problem. The makefile I was using to control the build on Linux was stripping the python executable to make it smaller. However, stripping the python executable on MacOSX breaks it completely so it can't load its shared libraries and I get errors as in my original message. If I remove the "strip" operation, then everything starts to work as expected. -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.x on MacOSX: failed dlopen() on .so's
On Wed, 2013-11-13 at 23:06 -0800, Ned Deily wrote: > On Nov 13, 2013, at 17:24 , Paul Smith wrote: > > I'm discovering that this is tricky. I don't want to bring OS wars into > > it, but this kind of thing is so simple and just works on GNU/Linux. I > > guess I've been spoiled :-). > > Well, if you are building from scratch, you would likely run into some > of the same issues on Linux systems, too. There are many variations > in configurations that are mutually incompatible. Possibly. However as long as you're using basic system libraries and not higher-level stuff like GTK etc., you can compile on an older version of Linux (say, Red Hat EL 5 or so) and the result will run without any problems on pretty much any distribution, even the newest ones: Red Hat, Ubuntu, Debian, Suse, etc. etc. We do this regularly, for production code we ship to customers. About the only "basic" libraries that cause problems are libssl/libcrypto... the openssl folks just don't seem to care much about this issue. > > I'm somewhat dreading my next effort after MacOS: the same thing, on > > Windows :-/. > > > > And another task, which seems like it will be fun: building GDB on > > MacOSX with Python support enabled... > > Keep in mind that on newer OS X releases, Apple no longer ships gcc or > gdb. The standard compiler is clang and it has its own debuggers. Yes, I'm aware. Very frustrating. We have a large set of specialized macros, both in native GDB and in Python using GDB's extension, that are invaluable for debugging our code. I'm not thrilled about the idea of trying to get all that working in lldb as well. So we're going to stick with GDB, building it ourselves, until it no longer works at all. At that point we'll decide how to proceed. > > Hm, that's an idea. I don't HAVE to build Python myself, actually, I > > just need (a) it to be relocatable, and (b) to add these extra modules > > to it so I can use it across systems without installing them > > individually by hand. > > I'm not sure I know what you mean by "relocatable". Like the > third-party package manager solutions, the python.org installers place > the Pythons they install in fixed locations (under /Library/Frameworks > with links from /usr/local/bin) but one that does not conflict with > any system files, including the Apple-supplied system Pythons. By relocatable I mean "runnable from any location"; i.e., not fixed. I have a wrapper around the Python executable that can compute the correct root directory and set any environment variables or add flags or whatever might be needed. Basically I have a large number of test systems and a set of test suites that are all written in Python, and I need to be able to distribute the same version of Python plus a specific set of additional modules across all those systems to be sure they all have the same environment. Plus the set of systems changes (new systems added/old ones removed) regularly. I'm handling this by checking in the Python distribution plus modules into a "tools" Git repository, then cloning it on each system. However, I have no special privileges on these systems (so I can't modify any system locations such as /usr/local) and I can't control what user account will be running the tests (so the user's home directory, where the tools repository is cloned, is not constant across the systems). I have this working on Linux with very little effort. Now I'm trying to get the same result on MacOSX. -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.x on MacOSX: failed dlopen() on .so's
On Wed, 2013-11-13 at 16:00 -0800, Ned Deily wrote: > > The reason I've set PYTHONHOME is ultimately I need this installation to > > be relocatable. It's going to be shared across lots of different > > systems and they'll have the ability to copy it wherever they want. > > That could be problematic. You need to be *really* careful about how > you do that. You stand a chance with a non-shared installation. You mean, --disable-shared? That's what I want, ultimately, but I was going to start with the default configuration first. I'm discovering that this is tricky. I don't want to bring OS wars into it, but this kind of thing is so simple and just works on GNU/Linux. I guess I've been spoiled :-). I'm somewhat dreading my next effort after MacOS: the same thing, on Windows :-/. And another task, which seems like it will be fun: building GDB on MacOSX with Python support enabled... > You still should not need to set PYTHONHOME. Also, be aware that > executables and libraries built on one version of OS X are not > guaranteed to work on other versions, particularly older versions > unless you take certain precautions. Even non-shared Pythons on OS X > dynamically link with system-supplied libraries which can vary across > os releases. And not all libraries are supplied, so, depending on > your needs, you may need to supply some additional third-party > libraries. This is why I'm building on this very old system, and am loath to update it. One saving grace is that while I need my installation to be relocatable, I _don't_ need it to be infinitely portable across MacOSX systems. I'm using it internally only and so I have some control over the version of MacOS and the hardware that it's running on. I don't need to worry about non-Intel hardware, or versions of MacOS prior to the one I'm using here. > For the python.org OS X binary installers, we go to a fair amount of > trouble to build Pythons that will work across a range of OS X > releases. You might want to consider using one of them as a base. > It's usually a lot less work than trying to make it work yourself. Hm, that's an idea. I don't HAVE to build Python myself, actually, I just need (a) it to be relocatable, and (b) to add these extra modules to it so I can use it across systems without installing them individually by hand. Thanks, I'll look into this further. -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.x on MacOSX: failed dlopen() on .so's
Thanks for the response Ned! On Wed, 2013-11-13 at 14:40 -0800, Ned Deily wrote: > There shouldn't be any problems with what you are trying to do. It > works for me with Python 2.7.6 and pycrypto-2.6.1. Some suggestions: > - Avoid --enable-shared on OS X at least initially. There are too > many ways things can go wrong. If you've built with it, suggest > starting with a fresh Python source directory just to be sure. I've tried it with all three options: no flag, --disable-shared, and --enable-shared, and don't notice any difference in the results; even with --enable-shared I don't seem to get any shared libs created. > - Check the dynamic library dependencies of _struct. On OS X: > > otool -L /Users/build/python/lib/python2.7/lib-dynload/_struct.so I get a libgcc_s reference as well, since I'm building with GCC: /Users/build/python/lib/python2.7/lib-dynload/_struct.so: /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0) /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1105.0.0) I wonder if using the GCC linker etc. is part of the problem? > - Make sure you are *really* building pycrypto and friends with your > Python and not with some other one. I'm pretty sure but I'll triple-check. The reason I've set PYTHONHOME is ultimately I need this installation to be relocatable. It's going to be shared across lots of different systems and they'll have the ability to copy it wherever they want. > - Check your other environment variables and make sure you are not > setting any DYLD_ or LD_ env variables. Hm; I am setting LD_LIBRARY_PATH to find the Python .so files. Does python figure out where to look for them by itself? Thanks for this info; I'll get back to you. -- https://mail.python.org/mailman/listinfo/python-list
python 2.7.x on MacOSX: failed dlopen() on .so's
Hi all. I need to build my own version of Python on a MacOSX system,
and I can't seem to do it successfully. I need to build it with a
particular location, etc. and so I can't use Homebrew or whatever: I
need to compile it myself from the source tarball. I did look through
the Homebrew recipe and I don't see anything that seems like it would
help with my problem.
I'm using MacOSX 10.7.5 with xcode 4.1, containing gcc 4.2.1 / clang 2.1
(configure seems to choose gcc).
I've tried this with both Python 2.7.5 and 2.7.6. I get the tarball,
unpack it, then:
$ configure --prefix=/Users/build/python
$ make
$ make install
$ export PATH=/Users/build/python/bin:$PATH
$ export PYTHONHOME=/Users/build/python
$ python2.7 --version
Python 2.7.6
Simple scripts seem to work fine. But then I need to install various
packages; for example pycrypto. When I try to build that it fails to
build:
$ cd src/pycrypto-2.6 && python2.7 setup.py build
Traceback (most recent call last):
File "setup.py", line 45, in
import struct
File "/Users/build/python/lib/python2.7/struct.py", line 1, in
from _struct import *
ImportError: dlopen(/Users/build/python/lib/python2.7/lib-dynload/_struct.so,
2): Symbol not found: _PyObject_Free
Referenced from: /Users/build/python/lib/python2.7/lib-dynload/_struct.so
Expected in: dynamic lookup
This happens with every module that is not pure-Python: boto, paramiko,
etc.
Looking at the various python2.7/lib-dynload/*.so files with nm I see
that actually a number of them refer to an unresolved ("U") symbol
_PyObject_Free (and other _PyObject_* symbols). I can't find any shared
library being built by the python build (.dylib or .so or whatever);
there's just a libpython2.7.a file (I tried forcing --enable-shared on
the configure line and it didn't change anything).
I should say that I've used this same configuration and set of steps,
etc. on my GNU/Linux system and it worked perfectly.
--
https://mail.python.org/mailman/listinfo/python-list
Can sqlite read gzipped databases?
Hi,
I'd like to read a series of sqlite database files that have already been
gzipped and was wondering if this can be done on the fly. In other words, can
I avoid explicitly unzipping the file into another file, but instead get an
SQL connection to the zip file either directly (can't see an option to do
this) or to an object in memory resulting from unzipping, eg. (hypothetically);
import gzip
from sqlite3 import dbapi2 as sqlite
data = gzip.GzipFile('Mydbase.db.gz','r')
d = data.read()
cnx = sqlite.connect(d) # or .connect(data)
cur = cnx.cursor()
etc
The above of course doesn't work, but just to give you the idea.
Thanks,
Paul
--
http://mail.python.org/mailman/listinfo/python-list
