Thanks for the link. Unfortunately there was not a reference to a
completed package that actually did this. As in, I really do not want
to reinvent the wheel. Ugh, sorry, that's a pun in this context.
Here is a first shot at this, just installing a moderately complicated
package in a virtualenv and then reinstalling it in another
virtualenv. Extract and execinput are my own programs (from drm_tools
on sourceforge) but it is obvious from the context what they are
doing. The links had to be soft because linux does not actually allow
a normal user (or maybe even root) to make a hard link to a directory.
cd /usr/common/lib/python3.6/Envs
rm -rf ~/.cache/pip #make download clearer
python3 -m venv scanpy
source scanpy/bin/activate
python -m pip install -U pip #update 9.0.3 to 20.1.1
which python3 #using the one in scanpy
pip3 install scanpy
scanpy -h #seems to start
deactivate
rm -rf ~/.cache/pip #make download clearer
python3 -m venv scanpy2
source scanpy2/bin/activate
python -m pip install -U pip #update 9.0.3 to 20.1.1
export DST=/usr/common/lib/python3.6/Envs/scanpy/lib/python3.6/site-packages
export SRC=/usr/common/lib/python3.6/Envs/scanpy2/lib/python3.6/site-packages
ls -1 $DST \
| grep -v __pycache__ \
| grep -v scanpy \
| grep -v easy_install.py \
| extract -fmt "ln -s $DST/[1,] $SRC/[1,]" \
| execinput
pip3 install scanpy
#downloaded scanpy, "Requirement already satisfied" for all the others
#Installing collected packages: scanpy
# Successfully installed scanpy-1.5.1
scanpy -h #seems to start
deactivate
source scanpy/bin/activate
scanpy -h #seems to start (still)
deactivate
So that method seems to have some promise. It saved a considerable
amount of space too:
du -k scanpy | tail -1
457408 scanpy
du -k scanpy2 | tail -1
24900 scanpy2
However, two potential problems are evident on inspection.
The first is that when the 2nd scanpy installation was performed it
updated the dates on all the directories in $DST. A workaround would
be to copy all of those directories into the virtualenv temporarily,
just for the installation, and then remove them and put the links in
afterwards. That strikes me as awfully cludgy. Setting them read
only would likely break the install.
The second issue is that each package install creates two directories like:
llvmlite
llvmlite-0.33.0.dist-info
where the latter contains top_level.txt which in turn contains one line:
llvmlite
pointing to the first directory.
If another version must cohabit with it the "llvmlite" directories
will conflict. For this sort of approach to work easily the llvmlite
directory should be named "llvmlite-0.33.0" and top_level.txt should
reference that too. It would be possible (probably) to work around it
though by having llvmlite-0.33.0 only in the common area and use:
ln -s $COMMON/llvmlite-0.33.0 $VENVAREA/llvmlite
The top_level.txt in each could then reference the unversioned name.
Unknown if this soft link approach will work on Windows.
Regards,
David Mathog
On Wed, Jun 24, 2020 at 1:26 PM Steve Dower <steve.do...@python.org> wrote:
>
> On 24Jun2020 1923, David Mathog wrote:
> > I think I will experiment a little with pipenv and if necessary after
> > each package install use a script to remove the installed libraries
> > and replace them with a hard link to the one in the common area.
> > Maybe it will be possible to put in those links before installing the
> > package of interest (like for scanpy, see first post), which will
> > hopefully keep it from having to rebuild all those packages too.
>
> Here's a recent discussion about this exact idea (with a link to an
> earlier discussion on this list):
> https://discuss.python.org/t/proposal-sharing-distrbution-installations-in-general/2524
>
> It's totally possible, though it's always a balance of trade-offs. Some
> of the people on that post may be interested in developing a tool to
> automate parts of the process.
>
> Cheers,
> Steve
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at
https://mail.python.org/archives/list/distutils-sig@python.org/message/2EMFGUE6QDTWBLPWDPE2TTOZOX3OFAOA/
