[Distutils] Fwd: Re: Use of "python" shebang an installation error?

2020-07-21 Thread David Mathog
(oops, had to resend, forgot to change the destination to
)

On Mon, Jul 20, 2020 at 12:38 PM John Thorvald Wodder II
 wrote:
>
> On 2020 Jul 20, at 15:25, David Mathog  wrote:
> > Lately I have been working on a CentOS 8 machine, and it has "python2"
> > and "python3", but no "python".  Many packages install scripts with a
> > shebang like:
> >
> >   #!/usr/bin/env python
> >
> > and those do not work on this OS.  Seems like rather a large missing
> > dependency which goes by without triggering a fatal error.
>
> How exactly are these packages getting installed?  Last time I checked, both 
> pip and setuptools automatically set the shebang in scripts (both 
> console_script entry points and scripts declared with the "scripts" argument 
> to `setup()`) to use the path of the running Python interpreter.  Are these 
> packages installed using your system package manager?  If so, you should take 
> the problem up with its maintainers.

Good point, I have been installing so many packages I get confused
about which installer was used for which package.   It turned out that
many (but not all) of the files which contained

   #!/usr/bin/env python

shebangs were installed using standard OS level tools (cmake,
configure, make and the like).  Example package, hisat2. I guess there
isn't much choice for those but to scan the directories for python
scripts and fix the shebangs.

Installs that are initially into venvs and used pip3 are still an
issue.  Example:

python3  -m venv johnnydep
cd johnnydep
grep -r '/usr/bin/env python$' .
#finds:
./lib/python3.6/site-packages/pip/_vendor/appdirs.py:#!/usr/bin/env python
./lib/python3.6/site-packages/pip/_vendor/chardet/cli/chardetect.py:#!/usr/bin/env
python
./lib/python3.6/site-packages/pip/_vendor/requests/certs.py:#!/usr/bin/env
python
./lib/python3.6/site-packages/pkg_resources/_vendor/appdirs.py:#!/usr/bin/env
python
./lib/python3.6/site-packages/johnnydep/pipper.py:#!/usr/bin/env python
cd bin
ls -1 | grep python
lrwxrwxrwx. 1 modules modules7 Jul 20 14:09 python -> python3
lrwxrwxrwx. 1 modules modules   16 Jul 20 14:09 python3 -> /usr/bin/python3
source activate
pip3 install johnnydep
head -1 johnnydep
#!/home/common/lib/python3.6/Envs/johnnydep/bin/python
#same for "tabulate" and all other shebangs in bin.
cd ..
grep -r '/usr/bin/env python$' .
#same as before
grep -r '/home/common/lib/python3.6/Envs/johnnydep/bin/python' .
#just the files in the bin directory.

It looks like none of the "#!/usr/bin/env python" shebangs within the
venv are going to be used after the install, so perhaps those are
harmless.

The shebangs like

   #!/home/common/lib/python3.6/Envs/johnnydep/bin/python

are OK within the venv, but once they are "devirtualized" they become
a problem.  That was a known problem though - my devirtualizer code
already patches all of the ones in the bin directory.  I have not seen
any elsewhere (yet) within the venv, but there is probably no rule
that keeps them from appearing in "share" or elsewhere.

The "python" in use in the venv is just a symbolic link to "python3"
which is itself a symbolic link to the actual program
"/usr/bin/python3". It is constructed that way based on "python -m
venv" which uses pieces which come from the CentOS 8
python3-libs-3.6.8-23.el8.x86_64 RPM.  Is there some requirement that
a venv have a "python"?  Odd that RedHat (and so CentOS) provide a
"python" there, but not in the OS itself.

Thanks,

David Mathog
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/XI7PKGVQFW63ZFMWMLR554VPAGWDHWZ6/


[Distutils] Fwd: Re: Use of "python" shebang an installation error?

2020-07-21 Thread David Mathog
On Mon, Jul 20, 2020 at 4:40 PM John Thorvald Wodder II
 wrote:
>
> First of all, your last two messages only went to me, not to the list.  The 
> mailing list doesn't set Reply-To on messages or the like, so you have to 
> manually set "To: distutils-sig@python.org" when replying.

Aargh, right.  I use gmail for my home mail, and since I'm stuck
working at home, that is what I used here.  Gmail likes to hide, well,
pretty much everything.  I will repost those responses.

>
> As to your e-mail, though, are any of those files even meant to be executed?  
> They're not in bin/; they just appear to be regular source files that some 
> developer slapped a shebang on.

That in a sense is the issue.  I don't know, you don't know, maybe the
developer knows (if he/she still remembers).  I really don't want to
do the work to dig through the code for every package I install to
determine if a shebang is used or not.  Yet if I don't figure this out
some end user will run a script (one of a hundred in some package I
installed for their use) which will blow up because of this issue.

The best I can do now is run

   pdvctrl reshebang $TARGET_DIR

or

   pdvctrl reshebang $ROOT_DIR...

and fix them up after the fact.  (pdvctrl from python_devirtualizer here:
https://sourceforge.net/projects/python-devirtualizer/).  Even then it
usually has to guess that "python" means "python3" and not "python2",
and sometimes it guesses wrong.  Today's version of that recurring
issue:

  https://github.com/lastz/lastz/issues/30

Regards,

David Mathog
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/LRLC63NI26ZGSCPIOYPA4MCKSM6JHXKS/


[Distutils] Fwd: Re: Use of "python" shebang an installation error?

2020-07-21 Thread David Mathog
(oops, had to resend, forgot to change the destination to
)

biopython-1.77, for instance, when installed into a virtualenv with
pip3, has many of these shebangs:

   #!/usr/bin/env python

And they are all over the place.  They are:

./site-packages/Bio/bgzf.py:
./site-packages/Bio/PDB/parse_pdb_header.py:
./site-packages/Bio/PDB/PDBList.py:
./site-packages/Bio/Restriction/__init__.py:
./site-packages/Bio/Restriction/Restriction.py:
./site-packages/Bio/Restriction/PrintFormat.py:
./site-packages/Bio/Restriction/Restriction_Dictionary.py:
./site-packages/Bio/Wise/__init__.py:
./site-packages/Bio/Wise/psw.py:
./site-packages/Bio/Wise/dnal.py:
./site-packages/Bio/UniProt/GOA.py:
./site-packages/Bio/SeqUtils/__init__.py:

Regards,

David Mathog
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/EPFOIYHVK62GYUWUVFMFGYFHZZDOHVRW/