[issue18309] Make python slightly more relocatable
Mathias Fröhlich added the comment:
Hey,
Shakeeb Alireza is right, the original problem was an application that links
and embeds against libpython*{so,dll,dynlib} and should more easily find
components like /*/lib*/python*.*/site.py and most probably now it needs to
find getpath.py as well.
While I am no longer working on that application where I wanted to have that
feature, I still believe it would be worthwhile to find the *.py files in the
file system relative to the location of the libpython*{so,dll,dynlib} file.
Thanks for taking care.
Mathias
--
___
Python tracker
<https://bugs.python.org/issue18309>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18309] Make python slightly more relocatable
Mathias Fröhlich added the comment: Ok, so far. But what shall I do now? It would be nice that python is a bit smarter in finding its increasing important module files when being embedded into an application. Anybody out there who wants to look at that contribution? best Mathias -- ___ Python tracker <https://bugs.python.org/issue18309> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18309] Make python slightly more relocatable
Mathias Fröhlich added the comment: Yes. msg191944 from Nick Coghlan, made me think that with all the initialization rework that appeared to be underway you want to incorporate that presented idea of basing the default onto the location of the libpython.so or the pythonX.X.dll instead of the location of python/python.exe. And as mentioned by Ronald Oussoren that would even align the methods used across the architectures to something common with a fallback to the current way that takes the path of the interpreter executable. At least that is what the provided patch implemented in the old code structure. And this does not even change the default for the common case where the default is plain useful. It is just changing the way how the default is determined so that the default for the case of an embedded interpreter is more meaningful. As stated somewhere above. The you can do that with application code when setting up the embedded interpreter, but it would be nice if that just works out of the box and by that helps applications not thinking of that solution. best Mathias -- ___ Python tracker <https://bugs.python.org/issue18309> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18309] Make python slightly more relocatable
Mathias Fröhlich added the comment: Hi, Nice to see some progress. Still, I checked todays https://github.com/python/cpython.git master and 3.8 branch (is that the current cpython development code?). Neither of them contain a call to dladdr beside the macos code path mentioned in msg191994 by Ronald Oussoren which does this already for a long time. By the lack of dladdr, I conclude that the code idea of my request here is not solved. May be to rephrase that. The basic idea behind that request was to make pythons default way to setup the paths required to find the python modules based on the place where the python library resides instead of the python executable program. I do not mean the compile time prefix but the actual location of the shared object in the file system. That would help to build applications that embed cpython, ship and unpack the whole application tree including the python modules to a custom location, while still preserving the subtree structure containing the python shared library and the python modules, not known at compile time. Note that this patch contained code to make that work from within python without custom code in the embedding application. Doing that on the embedding and calling application side was always possible and still is possible - but that was not the point. best Mathias -- resolution: fixed -> status: closed -> open ___ Python tracker <https://bugs.python.org/issue18309> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18309] Make python slightly more relocatable
Mathias Fröhlich added the comment: Hi Ronald, Eric, Nick, Looking up the symbol name of the current function should work also. And I am free to rename these functions to whatever you like. Attached is version 2 of the patch with the suggested changes. The windows implementation is still untested. It would be interesting to know if this kind of lookup scheme can be included into PEP 432. Provided the spirit of this PEP, I can imagine to provide several functions to build up the pythonpath starting from something. So say, have a 'get python path from argv[0]', a 'get python path from shared python library' and a 'get python path from prefix' function (I may miss a variant). Also a 'build python path from python home root entry point' function would be useful and could be used by the above functions. An application embedding python can then call those functions that are sensible for its own use and installation scheme to set the module path in the PyConfig struct. The Py_ReadConfig function will internally use the above suggested functions to build up the default configuration if not already provided. Greetings Mathias -- Added file: http://bugs.python.org/file30768/python-relative-path-lookup-v2.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18309 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18309] Make python slightly more relocatable
Mathias Fröhlich added the comment: Hi Eric, Thanks for looking at that ticket so fast! Reassigning this to 3.4 is great. In general, yes I can already do what I need more or less. This is the reason why I can be fine with about every python version. The point I bring up this change that I believe I am doing this at an unappropriate place as I need to know some internals of python when I do so and that I think that other can probably also benefit from this idea/change. What I currently do is to write an application that just uses python*.so as an embedded interpreter and this precompiled application might be relocated to about everywhere - just where it is unpacked. We are currently using the same sort of code to find out where the python*so file is and we use Py_SetPythonHome to set is to the directory where the so file resides. Why are we doing this? So, it takes the idea that is currently in the standard python interpreter. This one tries to be relocatable (means: pack the installation directory and unpack that somewhere else and be still able to run) by looking at argv[0] and dereferencing symbolic links until it arrives at a real file. Now suppose you want to embed python, then you do no longer use the standard python interpreter program. You may also use a different installation layout for basic things like bin and lib. So you end up with an application that is no longer able to find its provided python modules by looking at the applications path. But instead of starting from the path of the interpreter (which is not used in this case) or the application itself you could start from the python library path and look for your python installation relative to that. So as long as you stick with the relative file layout of everything that is python related (and only what is python related, the python.so and the modules) when you pack and unpack your precompiled application this would just work. So, put that in short: Instead of dynamically finding the the python module path relative to .../bin/python try to find the python relative to .../lib/libpython34.so. The benefit of that would be that every application that embeds python and needs to be relocatable will just work in the way that today only the standard python interpreter works. I try to get all of the PEP you pointed me to. As I am seeing this longer document the first time, I am not sure if I missed something there, but in that framework of this my proposal would probably influence the initial setting of sys.prefix (?) if this is not already provided from the embedding application. And yes I am perfectly fine with a different or more general approach. The initially attached patch is something that tried to integrate into the current checked in code as I understood it. Greetings Mathias -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18309 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18309] Make python slightly more relocatable
New submission from Mathias Fröhlich:
Hi all,
I want to move python a bit closer to be relocatable.
One problem to solve is where python finds its modules.
The usual lookup mechanism is to compile in a configure time
determined prefix that is used as a last resort path if the
paths are not set otherwise during application/interpreter startup.
The most commonly known way to change the module path at startup time
are probably the environment variables PYTHONPATH and PYTHONHOME.
The python interpreter itself already tries to interpret argv[0] to get to this
point, but it would be nice if an application embedded interpreter also finds
its module path without providing this argv[0] directly to the python library.
This should even work if being moved or being installed at a different path
than the configure time prefix path.
The proposal is to add an additional attempt to find the python modules
just before we resort to the compiled in prefix by looking at the path
to the python27.{so,dll}. Relative to this shared object python library
file the python modules are searched in the usual way. If there are
no python modules found relative to the python library file, the very
last resort compiled in prefix is used as usual.
For architectures where we cannot determine the path of the shared
library file, nothing changes.
I have attached a patch that tries to implement this.
It should serve as a base for discussions.
This change is tested on linux and behaves like expected. The windows code for
this is copied over from an other project where I have this actively running.
But this python code variant is not even compile tested on windows.
thanks in advance
Mathias
--
components: Installation
files: python-relative-path-lookup.diff
keywords: patch
messages: 191909
nosy: mathias
priority: normal
severity: normal
status: open
title: Make python slightly more relocatable
versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3, Python
3.4, Python 3.5
Added file: http://bugs.python.org/file30706/python-relative-path-lookup.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18309
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
