#20218: Use pip to install Python depedencies
---------------------------+-----------------------
Reporter: embray | Owner:
Type: enhancement | Status: new
Priority: minor | Milestone:
Component: build | Keywords:
Merged in: | Authors:
Reviewers: | Report Upstream: N/A
Work issues: | Branch:
Commit: | Dependencies:
Stopgaps: |
---------------------------+-----------------------
Work has already been done--for example in #19187--on using `pip` for
`sage -i` where appropriate. However, a normal build of sage does not use
`pip` for installing any Python spkgs; rather, they run `python setup.py
install` which until recently has always been *the* way to install Python
packages.
I can suggest several reasons why `pip` should be used instead of this
method. To be clear--as many people think of `pip` as a tool for
installing packages from the internet--this is not strictly the case.
`pip` can be used to install an already downloaded tarball, or even
straight from a source tree unpacked from a tarball or checked out from
VCS. The typical way to do this is to change directories to the directory
containing the `setup.py` script and running `pip install .`. This still
ultimately calls `setup.py install`, among other `setup.py` commands, but
with the right incantations to perform a pip-style flat install. Now, why
is this preferable?
1. Future-compatibility. One of the goals that the Python packaging
community has been working toward over the past several years has been to
decouple Python package installation from build system. Traditionally
`setup.py` provided both a build program and an installation program. At
best it will still stick around as a build program, but its use for
installation is being discouraged. There is now finally a PEP
(http://legacy.python.org/dev/peps/pep-0516/), still very much in draft
state, specifically concerning this decoupling. Under this new regime
`pip` will not likely be the *only* software for installing Python
packages, but the assumption can no longer be that there is a `setup.py`
script at all.
2. `pip` avoids using `easy_install`. `easy_install` is an older
installation program for Python packages that was bundled with setuptools
--when a Python project uses setuptools in its `setup.py`, the `setup.py
install` command invokes `easy_install` *by default*. The behavior of
`easy_install` is to install the package as an egg archive/directory.
`easy_install` has several disadvantages, of which this is just one, but
there are a couple reasons that this alone is a problem:[[br]][[br]]
a. Every .egg requires a `sys.path` entry, hence difficult to debug
paths that look like this:
{{{
#!python
>>> pprint(sys.path)
['',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/setuptools-18.1-py2.7.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/Cython-0.23.3-py2.7-linux-x86_64.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/MarkupSafe-0.23-py2.7-linux-x86_64.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/Jinja2-2.7.3-py2.7.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/six-1.10.0-py2.7.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/Pygments-2.0.2-py2.7.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/Sphinx-1.2.2-py2.7.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/zope.interface-4.1.3-py2.7-linux-x86_64.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/Twisted-15.5.0-py2.7-linux-x86_64.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages
/pytz-2013b0-py2.7.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/Babel-2.1.1-py2.7.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/Werkzeug-0.11.2-py2.7.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/speaklater-1.3-py2.7.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/python_openid-2.2.5-py2.7.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/itsdangerous-0.24-py2.7.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/Flask-0.10.1-py2.7.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/Flask_Silk-0.2-py2.7.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/Flask_AutoIndex-0.5-py2.7.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/Flask_Babel-0.9-py2.7.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/Flask_OpenID-1.2.5-py2.7.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/Flask_OldSessions-0.10-py2.7.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/webassets-0.11.1-py2.7.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/sagenb-0.11.6.1-py2.7.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/backports.ssl_match_hostname-3.4.0.2-py2.7.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/certifi-14.5.14-py2.7.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/tornado-4.1-py2.7-linux-x86_64.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/setuptools_scm-1.7.0-py2.7.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/path.py-7.1-py2.7.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/pickleshare-0.5-py2.7.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/simplegeneric-0.8.1-py2.7.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/decorator-4.0.6-py2.7.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/networkx-1.10-py2.7.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/python_dateutil-2.2-py2.7.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/cycler-0.9.0-py2.7.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/matplotlib-1.5.0-py2.7-linux-x86_64.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/jsonschema-2.4.0-py2.7.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/mistune-0.5.1-py2.7-linux-x86_64.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/pip-6.1.1-py2.7.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/pkgconfig-1.1.0-py2.7.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/singledispatch-3.4.0.3-py2.7.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-
packages/rpy2-2.7.5-py2.7-linux-x86_64.egg',
'/home/iguananaut/src/sagemath/sage/local/lib/python',
'/home/iguananaut/src/sagemath/sage/local/lib/python/site_packages',
'/home/iguananaut/src/sagemath/sage/local/lib/python27.zip',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/plat-linux2',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/lib-tk',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/lib-old',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/lib-dynload',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages']
}}}
With the flat install style used by `pip`, and now more generally
favored, the `sys.path`, with no eggs, would be simply:
{{{
#!python
>>> pprint(sys.path)
['',
'/home/iguananaut/src/sagemath/sage/local/lib/python',
'/home/iguananaut/src/sagemath/sage/local/lib/python27.zip',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/plat-linux2',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/lib-tk',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/lib-old',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/lib-dynload',
'/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages']
}}}
which is certainly easier to read and reason about.[[br]][[br]]
b. The egg import mechanism requires writing to a common file during egg
installation--easy-install.pth. This causes problems in parallel
installations, requiring patches to setuptools. The flat install style
used by pip has no such problems--there is a no single common file written
to by pip during installation of packages (other than maybe the log file,
which is only for debugging purposes).
As I explained in
[http://trac.sagemath.org/timeline?from=2016-02-25T10%3A23%3A37-08%3A00&precision=second
this comment], when a Python package is installed with `pip`, `pip`
actually slips setuptools into the installation process, whether the
package uses it explicitly or not. There are reasons for this not worth
going into now. This is no problem for the vast majority of cases. The
only exception I know of is that setuptools doesn't handle the
`data_files` option well when `--prefix` is changed. As far as I can tell
this is only a problem for sage itself and possibly for sagetex, but these
don't even use `setuptools` in the first place, so until we have a
workaround for that it does no harm if we keep using `setup.py install`
for those packages, if nothing else works.
Most of the rest of the Python packages included by default in a Sage
build (I haven't checked all of -optional yet) already use setuptools. I
checked the setup.py of all the rest that don't explicitly use setuptools
--these include `docutils`, `mpmath`, `pexpect`, `ptyprocess`, `Pycrypto`,
`pyparsing`, `pyzmq`, `scipy`, and all packages from the Jupyter team
(`ip*`, `jupyter*`, `np*`, `notebook`, `traitlets`).
Most of these packages are known by me (from personal experience) to work
fine with pip. Looking at the setup.py scripts of the rest of them, most
of them are fairly trivial and should work fine.
TL;DR, with the exceptions of sage and sagetex (pending work like #20108),
as well as pip itself the `spkg-install` scripts for Python packages
should change `setup.py install` commands to `pip install .`. Their
dependencies would also have to be updated to require pip to be installed
first.
At the same time the "uninstall" commands in those scripts can be changed
to use `pip uninstall`.
--
Ticket URL: <http://trac.sagemath.org/ticket/20218>
Sage <http://www.sagemath.org>
Sage: Creating a Viable Open Source Alternative to Magma, Maple, Mathematica,
and MATLAB
--
You received this message because you are subscribed to the Google Groups
"sage-trac" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/sage-trac.
For more options, visit https://groups.google.com/d/optout.
