On Thu, Oct 18, 2012 at 5:15 AM, Brian Harring <[email protected]> wrote: > If folks haven't looked at python_generate_wrapper_scripts in > python.eclass, I'd suggest doing so. For examples of it's usage, grep > for 'python_generate_wrapper_scripts' in /usr/bin/; any place you see > it, look for <that-script-name>-${PYTHON_TARGETS} (for example, > /usr/bin/sphinx-build{,-2.7,-3.2}. > > Each usage there is a separate custom script for that specific binary; > if there is a bug in the script, well, we're screwed- requires > re-merging the package. > > This setup, at least on my hardware, is .04s added to every > invocation; this is ignoring the inode cost for each, and the issue if > a bug ever appears in the script generation code (in which case we're > screwed- would require re-merging the package). > > In parallel, we've got python-wrapper (ls /usr/bin/python -l); this is > provided by eselect-python and basically discern what the active > python version is, and use that in the absense of any directives. > This is implemented in C, and is reasonably sane; the overhead for > that is basically non-existant. > > Roughly, I'm proposing we do away with python eclass's > generate_python_wrapper_scripts generation of a script, instead having > that just symlink to a binary provided by eselect-python that handles > this. This centralizes the implementation (fix in one spot), and > would allow a c version to be used- basically eliminating the > overhead. > > > There's a trick to this; currently, those generated scripts hardcode > the allowed/known python versions for that package. We obviously have > to preserve that; I propose we shove it into the symlink path. > > Basically, we add a /usr/libexec/python directory; within it, we have > a wrapper binary (explained below), and a set of symlinks pointing at > the root of that directory. To cover our current python versions, the > following would suffice: > > for x in {2.{4,5,6,7},3.{0,1,2,3,4}}-cpy 2.5-jython 2.7-pypy-1.{7,8} > \2.7-pypy-1.9; do > ln -s ./ /usr/libexec/python/$x > done > > While that seems insane, there is a reason; via that, we can encode > the allowed versions into the symlink. Using pkgcore's pquery for > example (which should support cpy: 2.5, 2.6, 2.7, 3.1, 3.2, 3.3) > instead of a wrapper script at /usr/bin/pquery, we'd have thus: > > targets=( 2.{5,6,7}-cpy 3.{1,2,3}-cpy ) > targets=$(IFS=/;echo -n "${targets[*]}") > # This results in > # targets=2.5-cpy/2.6-cpy/2.7-cpy/3.1-cpy/3.2-cpy/3.3-cpy > ln -s "/usr/libexec/python/${targets}/wrapper" \ > /usr/bin/pquery > > /usr/libexec/python/wrapper upon invocation, takes a look at argv[0]; > sees how it was invoked basically. This will be the /usr/bin/whatever > pathway. It reads the symlink, in the process getting the allowed > versions and preferred order of the versions. > > Few notes; vast majority of filesystems will store the symlink target > into the core inode if at all possible- in doing so, this avoids > wasting an inode and is only limited by the length of the target. > That length is capped by PATH_MAX- which can range from 256 to 4k (or > higher). > > For the pquery example above, that comes out to ~73 bytes for the > symlink pathway; well under PATH_MAX. > > For the scenarios where PATH_MAX caps the symlink pathway, or for > whatever reason we don't want to use that trick, a tree of files > contained within /usr/libexec/python/ holding the allowed versions for > the matching pathway would suffice. > > Either proposal here would be far faster than what we've got now; also > will use less space (ancillary benefit). > > One subtle aspect here is that if we did this, it makes it possible to > avoid losing the invocation information- currently if you did > `/usr/bin/python3.2 $(which sphinx-build) blah`, because of how things > are implemented now (specifically the two layers of wrappers)- you'll > get python2.7 running that sphinx-build invocation. > > This is wrong (it's directly discarding what the invocation > requested), although you're only going to see it for scripts that > do python introspection. > > Via doing the restructuring I'm mentioning above, that issue can be > fixed, while making things faster/saner. > > On a related note; we currently install multiple versions of the same > script- the only difference being the shebang. If one ignores the > shebang, in some cases this is necessary- where the script is 2to3 > translated, and the code for py2k vs py3k differs. For most, the only > difference is in the shebang however. > > While it's minor in space savings, it's possible to eliminate that > redundancy via a shebang target that looks at the pathway it was > invoked via. Fairly easy actually, and basically zero overhead if > done. > > Either way, thoughts? > > What I'm proposing isn't perfect, but I'm of the view it's a step up > from what's in place now- and via centralizing this crap, makes it > easier to change/maintain this going forward as necessary. > ~harring
If we are somehow going to eliminate the installation of a separate script for each python version, then the symlink idea sounds like a good solution for expressing the supported python versions. However, I'm really not sure how you would eliminate the separate scripts, in light of the 2to3 issue. Regarding your /usr/bin/python3.2 /usr/bin/sphinx-build example: invoking python on a binary (or a symlink to a binary) isn't going to work at all. So I don't see how you've solved that issue. Back to the discussion djc pointed out: mgorny has implemented a binary solution already, called "python-exec", in conjunction with python-r1.eclass. https://bitbucket.org/mgorny/python-exec python-exec assumes there will be a separate script for each version, and just calls exec several times. In shell code, it would look like this: scriptname=$0 # argv[0] exec scriptname-${EPYTHON} exec scriptname-$(< /etc/env.d/python/config) for x in python2.7 python2.6 python3.2 python3.1 ... ; do exec scriptname-${x} done The list of implementations for the loop at the end is hard-coded when python-exec is built. This is the weak point in the solution. It could be improved with your symlink idea. What are your thoughts on mgorny's python-exec solution? We could adapt it for python.eclass if you really want to support that. Probably by modifying python.eclass to install scripts like sphinx-python2.7 instead of sphinx-2.7.
