[Distutils] Re: package management - common storage while keeping the versions straight
On Tue, 2020-06-23 at 15:51 -0700, David Mathog wrote: > What I am after is some method of keeping exactly one copy of each > package-version in the common area (ie, one might find foo-1.2, > foo-1.7, and foo-2.3 there), while also presenting only the one > version of each (let's say foo-1.7) to a particular installed program. > On linux it might do that by making soft links to the common > PYTHONPATH area from another directory for which it sets PYTHONPATH > for the application. Finally, this has to be usable by any account > which has read execute access to the main directory. > > Does such a beast exist? If so, please point me to it! I have been meaning to do something like this for a while now! But unfortunately I can't find the time. If you do choose of start implementing it, please let me know. I would be happy to help out. Cheers, Filipe Laíns signature.asc Description: This is a digitally signed message part -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/ZIDMTET7JPOQJGCJR3L6EUDKOLGWYLRW/
[Distutils] Re: package management - common storage while keeping the versions straight
It turned out that the second install was not the cause of the timestamp change in the original. On reviewing "history" it turned out that I had accidentally run the link generation twice. That turned up this (for me) unexpected behavior: mkdir /tmp/foo ls -al /tmp/foo total 16 drwxrwxr-x. 2 modules modules 6 Jun 24 16:49 . drwxrwxrwt. 173 rootroot12288 Jun 24 16:49 .. ln -s /tmp/foo /tmp/bar ls -al /tmp/foo drwxrwxr-x. 2 modules modules 6 Jun 24 16:49 . drwxrwxrwt. 173 rootroot12288 Jun 24 16:49 .. ln -s /tmp/foo /tmp/bar ls -al /tmp/foo total 16 drwxrwxr-x. 2 modules modules17 Jun 24 16:51 . drwxrwxrwt. 173 rootroot12288 Jun 24 16:50 .. lrwxrwxrwx. 1 modules modules 8 Jun 24 16:51 foo -> /tmp/foo The repeated soft link actually put a file under the target. Strange. Apparently it is expected behavior. The problem can be avoided by using this form: ln -sn $TARGET $LINK The later installs are much faster than the first one, since putting in the links is very fast and building the packages is not. This was the trivial case though, since having done one install all the prerequisites were just "there". The johnnydep package will list the dependencies without doing the install. Guess I will throw something together based on that and the above results and see how it goes. Regards, David Mathog On Wed, Jun 24, 2020 at 4:23 PM Filipe Laíns wrote: > > On Tue, 2020-06-23 at 15:51 -0700, David Mathog wrote: > > What I am after is some method of keeping exactly one copy of each > > package-version in the common area (ie, one might find foo-1.2, > > foo-1.7, and foo-2.3 there), while also presenting only the one > > version of each (let's say foo-1.7) to a particular installed program. > > On linux it might do that by making soft links to the common > > PYTHONPATH area from another directory for which it sets PYTHONPATH > > for the application. Finally, this has to be usable by any account > > which has read execute access to the main directory. > > > > Does such a beast exist? If so, please point me to it! > > I have been meaning to do something like this for a while now! But > unfortunately I can't find the time. > > If you do choose of start implementing it, please let me know. I would > be happy to help out. > > Cheers, > Filipe Laíns -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/63NJKSY7BLPJZXLK5DJFWROGQUKJ7RVF/
[Distutils] Re: package management - common storage while keeping the versions straight
Thanks for the link. Unfortunately there was not a reference to a completed package that actually did this. As in, I really do not want to reinvent the wheel. Ugh, sorry, that's a pun in this context. Here is a first shot at this, just installing a moderately complicated package in a virtualenv and then reinstalling it in another virtualenv. Extract and execinput are my own programs (from drm_tools on sourceforge) but it is obvious from the context what they are doing. The links had to be soft because linux does not actually allow a normal user (or maybe even root) to make a hard link to a directory. cd /usr/common/lib/python3.6/Envs rm -rf ~/.cache/pip #make download clearer python3 -m venv scanpy source scanpy/bin/activate python -m pip install -U pip #update 9.0.3 to 20.1.1 which python3 #using the one in scanpy pip3 install scanpy scanpy -h #seems to start deactivate rm -rf ~/.cache/pip #make download clearer python3 -m venv scanpy2 source scanpy2/bin/activate python -m pip install -U pip #update 9.0.3 to 20.1.1 export DST=/usr/common/lib/python3.6/Envs/scanpy/lib/python3.6/site-packages export SRC=/usr/common/lib/python3.6/Envs/scanpy2/lib/python3.6/site-packages ls -1 $DST \ | grep -v __pycache__ \ | grep -v scanpy \ | grep -v easy_install.py \ | extract -fmt "ln -s $DST/[1,] $SRC/[1,]" \ | execinput pip3 install scanpy #downloaded scanpy, "Requirement already satisfied" for all the others #Installing collected packages: scanpy # Successfully installed scanpy-1.5.1 scanpy -h #seems to start deactivate source scanpy/bin/activate scanpy -h #seems to start (still) deactivate So that method seems to have some promise. It saved a considerable amount of space too: du -k scanpy | tail -1 457408 scanpy du -k scanpy2 | tail -1 24900 scanpy2 However, two potential problems are evident on inspection. The first is that when the 2nd scanpy installation was performed it updated the dates on all the directories in $DST. A workaround would be to copy all of those directories into the virtualenv temporarily, just for the installation, and then remove them and put the links in afterwards. That strikes me as awfully cludgy. Setting them read only would likely break the install. The second issue is that each package install creates two directories like: llvmlite llvmlite-0.33.0.dist-info where the latter contains top_level.txt which in turn contains one line: llvmlite pointing to the first directory. If another version must cohabit with it the "llvmlite" directories will conflict. For this sort of approach to work easily the llvmlite directory should be named "llvmlite-0.33.0" and top_level.txt should reference that too. It would be possible (probably) to work around it though by having llvmlite-0.33.0 only in the common area and use: ln -s $COMMON/llvmlite-0.33.0 $VENVAREA/llvmlite The top_level.txt in each could then reference the unversioned name. Unknown if this soft link approach will work on Windows. Regards, David Mathog On Wed, Jun 24, 2020 at 1:26 PM Steve Dower wrote: > > On 24Jun2020 1923, David Mathog wrote: > > I think I will experiment a little with pipenv and if necessary after > > each package install use a script to remove the installed libraries > > and replace them with a hard link to the one in the common area. > > Maybe it will be possible to put in those links before installing the > > package of interest (like for scanpy, see first post), which will > > hopefully keep it from having to rebuild all those packages too. > > Here's a recent discussion about this exact idea (with a link to an > earlier discussion on this list): > https://discuss.python.org/t/proposal-sharing-distrbution-installations-in-general/2524 > > It's totally possible, though it's always a balance of trade-offs. Some > of the people on that post may be interested in developing a tool to > automate parts of the process. > > Cheers, > Steve -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/2EMFGUE6QDTWBLPWDPE2TTOZOX3OFAOA/
[Distutils] Re: Twine 3.2.0 released
Congrats! And thanks for stepping up to maintain one of our important tools! On 24Jun2020 1114, Brian Rutledge wrote: https://pypi.org/project/twine/3.2.0/ Changelog: https://twine.readthedocs.io/en/latest/changelog.html This is my first release as a Twine maintainer. Thanks to: - Sumana Harihareswara for getting me started, and nudging me along the way - Ian Stapleton Cordasco, Jason R. Coombs, and Dustin Ingram for code review and guidance in general - Devesh Kumar Singh, Felipe Mulinari Rocha Campos, and Vikram Jayanthi for substantial contributions to this release -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/FYPVPW7UFQ2MZRHURLXCZRC2HOAG3726/
[Distutils] Re: package management - common storage while keeping the versions straight
On 24Jun2020 1923, David Mathog wrote: I think I will experiment a little with pipenv and if necessary after each package install use a script to remove the installed libraries and replace them with a hard link to the one in the common area. Maybe it will be possible to put in those links before installing the package of interest (like for scanpy, see first post), which will hopefully keep it from having to rebuild all those packages too. Here's a recent discussion about this exact idea (with a link to an earlier discussion on this list): https://discuss.python.org/t/proposal-sharing-distrbution-installations-in-general/2524 It's totally possible, though it's always a balance of trade-offs. Some of the people on that post may be interested in developing a tool to automate parts of the process. Cheers, Steve -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/IUGVEOU5U7VYJMEUC6E3VWBE6OFTLPHR/
[Distutils] Re: package management - common storage while keeping the versions straight
On Wed, Jun 24, 2020 at 1:36 AM Thomas Kluyver wrote: > > On Tue, 23 Jun 2020, at 23:51, David Mathog wrote: > > What I am after is some method of keeping exactly one copy of each > > package-version in the common area (ie, one might find foo-1.2, > > foo-1.7, and foo-2.3 there), while also presenting only the one > > version of each (let's say foo-1.7) to a particular installed program. > Conda environments work somewhat like this - all the packages are stored in a > central place, and the structure of selected ones is replicated using > hardlinks in a site-packages directory belonging to the environment. So if > your concern is not to waste disk space by storing copies of the same > packages, that might be an option. I experimented with that one a little. It installs its own copies of python and things like openssl and openblas which are already present from the linux distribution. Similarly, if some python script needs "bwa" it will install its own even though that program is already available. Basically it is yet another "replicate everything we might need whether or not it is already present" type of solution. (The extreme end of that spectrum are systems like docker, which effectively replaces the entire OS.) So there might be only the one version of each python package (not counting duplicates with the OS's python3) but now there are also duplicate copies of system libraries and utilities. I think I will experiment a little with pipenv and if necessary after each package install use a script to remove the installed libraries and replace them with a hard link to the one in the common area. Maybe it will be possible to put in those links before installing the package of interest (like for scanpy, see first post), which will hopefully keep it from having to rebuild all those packages too. Thanks, David Mathog -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/QBIRYI767AVZ2FCFHVTP56XIKOX4TTYQ/
[Distutils] Twine 3.2.0 released
https://pypi.org/project/twine/3.2.0/ Changelog: https://twine.readthedocs.io/en/latest/changelog.html This is my first release as a Twine maintainer. Thanks to: - Sumana Harihareswara for getting me started, and nudging me along the way - Ian Stapleton Cordasco, Jason R. Coombs, and Dustin Ingram for code review and guidance in general - Devesh Kumar Singh, Felipe Mulinari Rocha Campos, and Vikram Jayanthi for substantial contributions to this release -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/FQUG3CW2T5WPDXFLF6JSMTGOSRLXD6OJ/
[Distutils] Re: package management - common storage while keeping the versions straight
On Tue, 23 Jun 2020, at 23:51, David Mathog wrote: > What I am after is some method of keeping exactly one copy of each > package-version in the common area (ie, one might find foo-1.2, > foo-1.7, and foo-2.3 there), while also presenting only the one > version of each (let's say foo-1.7) to a particular installed program. > On linux it might do that by making soft links to the common > PYTHONPATH area from another directory for which it sets PYTHONPATH > for the application. Finally, this has to be usable by any account > which has read execute access to the main directory. Conda environments work somewhat like this - all the packages are stored in a central place, and the structure of selected ones is replicated using hardlinks in a site-packages directory belonging to the environment. So if your concern is not to waste disk space by storing copies of the same packages, that might be an option. Thomas -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/AIMQGCP5TSCYAC2DOGZMUQ36L3MZ7K55/
[Distutils] Re: package management - common storage while keeping the versions straight
On Wed, 24 Jun 2020 at 00:00, David Mathog wrote: > Does such a beast exist? If so, please point me to it! Basically no, or at least not to my knowledge. The mechanisms exist, in the form of import hooks and similar, to build something like this, but it's not proved to be a common enough requirement that there's a well-known/standard library for it. I believe that setuptools (pkg_resources) had a mechanism to do something along these lines, but it never really became popular and I don't know if it's still considered as supported by the setuptools maintainers. So I think you're going to have to either accept the need for multiple copies, or write something specific for your situation. Sorry, Paul -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/4P27HIKDPPEV5J4HQXW2D6N4VBCJJUUF/
