On 2020-07-21 22:50, David Mathog wrote:
(oops, had to resend, forgot to change the destination to
<distutils-sig@python.org>)
On Mon, Jul 20, 2020 at 12:38 PM John Thorvald Wodder II
<jwod...@gmail.com> wrote:
On 2020 Jul 20, at 15:25, David Mathog <dmat...@gmail.com> wrote:
Lately I have been working on a CentOS 8 machine, and it has "python2"
and "python3", but no "python". Many packages install scripts with a
shebang like:
#!/usr/bin/env python
and those do not work on this OS. Seems like rather a large missing
dependency which goes by without triggering a fatal error.
How exactly are these packages getting installed? Last time I checked, both pip and
setuptools automatically set the shebang in scripts (both console_script entry points and
scripts declared with the "scripts" argument to `setup()`) to use the path of
the running Python interpreter. Are these packages installed using your system package
manager? If so, you should take the problem up with its maintainers.
Good point, I have been installing so many packages I get confused
about which installer was used for which package. It turned out that
many (but not all) of the files which contained
#!/usr/bin/env python
shebangs were installed using standard OS level tools (cmake,
configure, make and the like). Example package, hisat2. I guess there
isn't much choice for those but to scan the directories for python
scripts and fix the shebangs.
Hi,
For many of these tools, you can pass in something like
PYTHON=/usr/bin/python3 at build or install time to select the interpreter.
Installs that are initially into venvs and used pip3 are still an
issue. Example:
python3 -m venv johnnydep
cd johnnydep
grep -r '/usr/bin/env python$' .
#finds:
./lib/python3.6/site-packages/pip/_vendor/appdirs.py:#!/usr/bin/env python
./lib/python3.6/site-packages/pip/_vendor/chardet/cli/chardetect.py:#!/usr/bin/env
python
./lib/python3.6/site-packages/pip/_vendor/requests/certs.py:#!/usr/bin/env
python
./lib/python3.6/site-packages/pkg_resources/_vendor/appdirs.py:#!/usr/bin/env
python
./lib/python3.6/site-packages/johnnydep/pipper.py:#!/usr/bin/env python
cd bin
ls -1 | grep python
lrwxrwxrwx. 1 modules modules 7 Jul 20 14:09 python -> python3
lrwxrwxrwx. 1 modules modules 16 Jul 20 14:09 python3 -> /usr/bin/python3
source activate
pip3 install johnnydep
head -1 johnnydep
#!/home/common/lib/python3.6/Envs/johnnydep/bin/python
#same for "tabulate" and all other shebangs in bin.
cd ..
grep -r '/usr/bin/env python$' .
#same as before
grep -r '/home/common/lib/python3.6/Envs/johnnydep/bin/python' .
#just the files in the bin directory.
It looks like none of the "#!/usr/bin/env python" shebangs within the
venv are going to be used after the install, so perhaps those are
harmless.
The shebangs like
#!/home/common/lib/python3.6/Envs/johnnydep/bin/python
are OK within the venv, but once they are "devirtualized" they become
a problem. That was a known problem though - my devirtualizer code
already patches all of the ones in the bin directory. I have not seen
any elsewhere (yet) within the venv, but there is probably no rule
that keeps them from appearing in "share" or elsewhere.
The "python" in use in the venv is just a symbolic link to "python3"
which is itself a symbolic link to the actual program
"/usr/bin/python3". It is constructed that way based on "python -m
venv" which uses pieces which come from the CentOS 8
python3-libs-3.6.8-23.el8.x86_64 RPM. Is there some requirement that
a venv have a "python"? Odd that RedHat (and so CentOS) provide a
"python" there, but not in the OS itself.
Yes, a venv will always have a "python".
For the OS itself, that was a deliberate (but tough) decision. It's not
easy to find out what "python" should mean in a particular script (as
you found out), and the errors you get when using the wrong one are not
always enlightening. For an OS released before end of life of Python 2,
but supported well after it, we* "refuse to guess" and don't provide
"python" by default.
In a venv, though, all is clear: you get the Python you created the venv
with.
There's some more info here:
- across Linux distros: https://www.python.org/dev/peps/pep-0394/
- for EL8: https://developers.redhat.com/blog/2018/11/14/python-in-rhel-8/
From elsewhere in the thread:
The best I can do now is run
pdvctrl reshebang $TARGET_DIR
or
pdvctrl reshebang $ROOT_DIR...
and fix them up after the fact. (pdvctrl from python_devirtualizer here:
https://sourceforge.net/projects/python-devirtualizer/). Even then it
usually has to guess that "python" means "python3" and not "python2",
and sometimes it guesses wrong.
My recommendation is to try not to guess: either provide an option, or
look at the Python in the venv you're devirtualizing.
Also note that a "reshebang" tool is included in Python source code[0],
but it's not part of the standard library, so it doesn't always come
with Python. (On Fedora/RHEL/Centos it's in the python3-devel package as
/usr/bin/pathfix.py). We* use is quite often, and found some edge cases
involving flags in shebangs. You might want to look into it if you run
into those for devirtualizer's reshebang.
* here, "we" == Red Hat's Python maintenance team
[0]: https://github.com/python/cpython/blob/master/Tools/scripts/pathfix.py
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at
https://mail.python.org/archives/list/distutils-sig@python.org/message/YHK37CXP6X4RR7CGYWS2F4TCCW2O3FG5/
