[Distutils] Re: pip install very slow
See https://github.com/pypa/pip/issues/7555 for details on this.
Stuart McGraw wrote:
“Hello,
First time trying to use Python's package tools and I'm attempting to use pip
to install from a local project:
python3 -m pip install --user --upgrade --no-deps ./
I have a simple directory structure very similar to that recommended in
https://packaging.python.org/tutorials/packaging-projects/
project/
data/
mypackage/
__init__.py
data/
*.csv
module1.py
module2.py
tmpl/
*.jinja
setup.py
The project/data/ directory contains several GB of stuff I use during
development. It is not supposed to be part of the package and it is not
mentioned in setup.py.
The problem is when I run the install command above it takes many minutes to
finish. When I move the project/data/ sub-directory out of the project/
directory, it takes a few seconds.
I thought perhaps that "packages=setuptools.find_packages()" in my setup.py
file might be the culprit but replacing in with "packages=['mypackage']" made
no difference.
I'm not sure why pip seems to be roaming through project/data/ (if that's
what's happening) but is there some way to get it not to? My apologies if this
is covered in the docs somewhere; I'm still going though them but didn't see
anything with an initial scan.
If it helps, here is my setup.py file:
setuptools.setup (
name="mypackage",
version="...",
author="Me",
author_email="...",
description="...",
long_description="...",
long_description_content_type="text/markdown",
url="...",
packages=['mypackage'],
package_data={'mypackage': ['data/*.csv', 'tmpl/*.jinja',]},
zip_safe = False,
classifiers=[...],
python_requires='>=3.6',
install_requires = ['jinja2',], )
Thanks for any enlightenment.
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at
https://mail.python.org/archives/list/distutils-sig@python.org/message/4JVTE7ZEGR4G3L6ISCT6USFQCFJZEAVA/”
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at
https://mail.python.org/archives/list/distutils-sig@python.org/message/W4AJNOYB6W6F6PLP2DWJ5WEJYLFBR2A2/
[Distutils] pip install very slow
Hello,
First time trying to use Python's package tools and I'm attempting to use pip
to install from a local project:
python3 -m pip install --user --upgrade --no-deps ./
I have a simple directory structure very similar to that recommended in
https://packaging.python.org/tutorials/packaging-projects/
project/
data/
mypackage/
__init__.py
data/
*.csv
module1.py
module2.py
tmpl/
*.jinja
setup.py
The project/data/ directory contains several GB of stuff I use during
development. It is not supposed to be part of the package and it is not
mentioned in setup.py.
The problem is when I run the install command above it takes many minutes to
finish. When I move the project/data/ sub-directory out of the project/
directory, it takes a few seconds.
I thought perhaps that "packages=setuptools.find_packages()" in my setup.py file might be
the culprit but replacing in with "packages=['mypackage']" made no difference.
I'm not sure why pip seems to be roaming through project/data/ (if that's
what's happening) but is there some way to get it not to? My apologies if this
is covered in the docs somewhere; I'm still going though them but didn't see
anything with an initial scan.
If it helps, here is my setup.py file:
setuptools.setup (
name="mypackage",
version="...",
author="Me",
author_email="...",
description="...",
long_description="...",
long_description_content_type="text/markdown",
url="...",
packages=['mypackage'],
package_data={'mypackage': ['data/*.csv', 'tmpl/*.jinja',]},
zip_safe = False,
classifiers=[...],
python_requires='>=3.6',
install_requires = ['jinja2',], )
Thanks for any enlightenment.
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at
https://mail.python.org/archives/list/distutils-sig@python.org/message/4JVTE7ZEGR4G3L6ISCT6USFQCFJZEAVA/
[Distutils] Re: package management - common storage while keeping the versions straight
On Thu, Jun 25, 2020 at 12:37 AM Paul Moore wrote:
> I think the key message here is that you won't be *re*-inventing the
> wheel. This is a wheel that still needs to be invented.
It _was_ invented, but it is off round and gives a rough ride. As
noted in the first post this:
__requires__ = ['scipy <1.3.0,>=1.2.0', 'anndata <0.6.20', 'loompy
<3.0.0,>=2.00', 'h5py <2.10']
import pkg_resources
was able to load the desired set of package-versions for scanpy, but
setting a version number constraint on scanpy itself at the end of
that list, one which matched the version that the preceding commands
successfully loaded, broke it. So it is not reliable.
And the entire __requires__ kludge is only present because for reasons
beyond my pay grade this:
import pkg_resources
pkg_resources.require("scipy<1.3.0,>=1.2.0;anndata<0.6.20;etc.")
import scipy
import anndata
#etc.
cannot work because by default "import pkg_resources" keeps only the
most recent version rather than making up a tree (or list or hash or
whatever) and waiting to see if there are any version constraints to
be applied at the time of actual package import.
What I'm doing now is basically duct tape and bailing wire to work
around those deeper issues. In terms of language design, a much
better fix would be to modify pkg_resources so that it will always
successfully load the required versions from a designated directory
which contains multiple versions of packages, and modify the package
maintenance tools so that they can maintain such a directory.
Regards,
David Mathog
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at
https://mail.python.org/archives/list/distutils-sig@python.org/message/X23JIVPWU74HW3GBMVJEKAC2XUFROKAL/
[Distutils] Re: package management - common storage while keeping the versions straight
On Tue, 2020-06-23 at 15:51 -0700, David Mathog wrote: > What I am after is some method of keeping exactly one copy of each > package-version in the common area (ie, one might find foo-1.2, > foo-1.7, and foo-2.3 there), while also presenting only the one > version of each (let's say foo-1.7) to a particular installed program. > On linux it might do that by making soft links to the common > PYTHONPATH area from another directory for which it sets PYTHONPATH > for the application. Finally, this has to be usable by any account > which has read execute access to the main directory. > > Does such a beast exist? If so, please point me to it! zc.buildout[1] and zc.recipe.egg[2] can do something very much like this. zc.buildout tries hard to maintain reproducibility and isolation, and one of the ways it does this is by keeping each package in its own .egg directory. It then generates the entry-point scripts and a REPL with ``sys.path`` explicitly set to reference exactly the versions specified. There's no virtualenv-like activation step that puts all those scripts on the path, though, so one must either do that manually or just invoke the generated scripts directly. For example, here's a buildout configuration that specifies a shared directory to store eggs in. It also has some parts using zc.recipe.egg, one that will use zope.interface 4 and one that will use zope.interface 5 (egg specifications can be arbitrarily complex, of course, dependencies are followed, etc): [buildout] eggs-directory = //buildout-eggs abi-tag-eggs = true parts = old-interface new-interface new-interface-plus [old-interface] recipe = zc.recipe.egg eggs = zope.interface == 4.0 interpreter = old-py [new-interface] recipe = zc.recipe.egg eggs = zope.interface == 5.1 interpreter = new-py [new-interface-plus] recipe = zc.recipe.egg eggs = zope.interface == 5.1 zope.component interpreter = new-py-plus After running `buildout`, I have three REPL files (if zope.interface had defined any entry point scripts, I could also have had those generated): $ head bin/new-py #! import sys sys.path[0:0] = [ '//buildout-eggs/pypy_73/zope.interface-5.1.0-py2.7.egg', '//site-packages', ] $ head bin/old-py #! import sys sys.path[0:0] = [ '//buildout-eggs/pypy_73/zope.interface-4.0.0-py2.7.egg', '//site-packages', ] $ head bin/new-py-plus #! import sys sys.path[0:0] = [ '//buildout-eggs/pypy_73/zope.interface-5.1.0-py2.7.egg', '//buildout-eggs/pypy_73/zope.component-4.6.1-py2.7.egg', '//site-packages', ] I've got a collection of zope.interface eggs referenced from a variety of different buildouts (at least at one point in time) and from a variety of different Python implementations, but only ever one copy of each: $ ls -ld buildout-eggs/*/zope.interface* Permissions Size UserDate ModifiedName drwxr-xr-x - jmadden 2017-05-04 06:53 buildout-eggs/pypy_41/zope.interface-4.4.0-py2.7.egg/ drwxr-xr-x - jmadden 2017-05-04 07:00 buildout-eggs/cp27m/zope.interface-4.4.0-py2.7-macosx-10.12-x86_64.egg/ drwxr-xr-x - jmadden 2017-05-09 17:55 buildout-eggs/cp34m/zope.interface-4.4.0-py3.4-macosx-10.12-x86_64.egg/ drwxr-xr-x - jmadden 2017-06-08 10:20 buildout-eggs/cp27m/zope.interface-4.4.1-py2.7-macosx-10.12-x86_64.egg/ drwxr-xr-x - jmadden 2017-07-11 10:07 buildout-eggs/cp36m/zope.interface-4.4.2-py3.6-macosx-10.12-x86_64.egg/ drwxr-xr-x - jmadden 2017-12-08 11:10 buildout-eggs/cp27m/zope.interface-3.6.7-py2.7-macosx-10.13-x86_64.egg/ drwxr-xr-x - jmadden 2018-05-07 11:05 buildout-eggs/cp27m/zope.interface-4.1.3-py2.7-macosx-10.13-x86_64.egg/ drwxr-xr-x - jmadden 2020-03-13 07:53 buildout-eggs/cp27m/zope.interface-4.6.0-py2.7-macosx-10.15-x86_64.egg/ drwxr-xr-x - jmadden 2020-04-08 07:32 buildout-eggs/cp38/zope.interface-5.1.0-py3.8-macosx-10.15-x86_64.egg/ drwxr-xr-x - jmadden 2020-05-17 09:30 buildout-eggs/cp27m/zope.interface-5.1.0-py2.7.egg/ drwxr-xr-x - jmadden 2020-06-11 07:42 buildout-eggs/pypy_73/zope.interface-5.1.0-py2.7.egg/ drwxr-xr-x - jmadden 2020-06-25 11:58 buildout-eggs/pypy_73/zope.interface-4.0.0-py2.7.egg/ ~Jason [1] http://www.buildout.org/ ; I recommend version 3, which uses pip to find eggs [2] https://pypi.org/project/zc.recipe.egg/ -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/G52C7F6QHH3TINSOYBMI5CV54SWCZMPT/
[Distutils] Re: package management - common storage while keeping the versions straight
On Thu, 25 Jun 2020 at 00:06, David Mathog wrote: > > Thanks for the link. Unfortunately there was not a reference to a > completed package that actually did this. As in, I really do not want > to reinvent the wheel. Ugh, sorry, that's a pun in this context. I think the key message here is that you won't be *re*-inventing the wheel. This is a wheel that still needs to be invented. Paul (It was *way* too hard trying to write the above without tripping over the extended "wheel" pun ;-)) -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/JHCOMW4KYYKNHDA5KDNAMOS33OWM5BSM/
