Hi all,
First post here.
I have a cluster where the common software is NFS shared from the file
server to other nodes. All the python packages are kept in a
directory which is referenced by PYTHONPATH. The good part of that is
that there is just one copy of each package-version. The bad part, as
you have all no doubt guessed, is that python by itself is really bad
at specifying and loading a set of particular library versions (see
below), so upgrading one program will break another due to conflicting
installed versions. Hence the common use of virtualenv's. But as far
as I can tell each virtualenv installs a copy of each package-version
it needs, resulting in multiple copies of the same package-version for
common packages on the same disk.
What I am after is some method of keeping exactly one copy of each
package-version in the common area (ie, one might find foo-1.2,
foo-1.7, and foo-2.3 there), while also presenting only the one
version of each (let's say foo-1.7) to a particular installed program.
On linux it might do that by making soft links to the common
PYTHONPATH area from another directory for which it sets PYTHONPATH
for the application. Finally, this has to be usable by any account
which has read execute access to the main directory.
Does such a beast exist? If so, please point me to it!
The limitations of python version handling to which I refer above can
be illustrated for "scanpy-scripts"'s dependencies. Given all the
needed libraries in one place (plus incompatible versions) the right
set can be loaded (and verified) like this:
export PYTHONPATH=/path/to_common_area
python3
__requires__ = ['scipy <1.3.0,>=1.2.0', 'anndata <0.6.20', 'loompy
<3.0.0,>=2.00', 'h5py <2.10']
import pkg_resources
import scipy
import anndata
import loompy
import h5py
import scanpy
print(scipy.__version__)
print(anndata.__version__)
print(loompy.__version__)
print(h5py.__version__)
print(scanpy.__version__)
quit()
which emits exactly the versions scanpy-scripts needs:
1.2.3
0.6.19
2.0.17
2.9.0
1.4.3
However, adding
, 'scanpy <1.4.4,>=1.4.2'
at the end of __requires__ makes the whole thing fail at
import pkg_resources
with
(many lines deleted)
792, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (scipy 1.2.3
(/home/common/lib/python3.6/site-packages/scipy-1.2.3-py3.6-linux-x86_64.egg),
Requirement.parse('scipy>=1.3.1'), {'umap-learn'})
even though the scanpy it loaded in the first case was within the
desired range. Moreover, specifying the desired versions as
parameters to
import pkg_resources
does not work at all since pkg_resources only keeps the highest
version of each package it finds when imported. (A limitation that
never made the least bit of sense to me.)
The test system is CentOS 8 with python 3.6.8.
Thanks,
David Mathog
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at
https://mail.python.org/archives/list/distutils-sig@python.org/message/C44E6LUGKGNKKXCEZJMOJUG3HMZKUYG2/
