[Distutils] Re: package management - common storage while keeping the versions straight

2020-07-07 Thread David Mathog
Hi all. "Python devirtualizer" is a preliminary implementation which manages shared packages so that only one copy of each package version is required. It installs into a virtualenv, then migrates the contents out into the normal OS environment, and while so doing, replaces what would be

[Distutils] Re: package management - common storage while keeping the versions straight

2020-06-29 Thread John Thorvald Wodder II
On 2020 Jun 29, at 16:09, David Mathog wrote: > > In neither case does the egg-info file reference the corresponding > directory, but at least the directory in both has the expected package > name (other than case). In the examples you cited at the top, were > any of those "different name"

[Distutils] Re: package management - common storage while keeping the versions straight

2020-06-29 Thread David Mathog
On Fri, Jun 26, 2020 at 2:51 PM John Thorvald Wodder II wrote: > Of the 32,517 non-matching projects, 7,117 were Odoo projects with project > names of the form "odoo{version}_addon_{foo}" containing namespace modules of > the form "odoo/addons/{foo}", and 3,175 were Django projects with

[Distutils] Re: package management - common storage while keeping the versions straight

2020-06-27 Thread Paul Moore
On Sat, 27 Jun 2020 at 01:37, David Mathog wrote: > > Thanks for that feedback. Looks like RECORD is the one to use. > > The names of the directories ending in dist-info seem to be uniformly: > > package-version.dist_info Note that if you're doing something like this, you should probably read

[Distutils] Re: package management - common storage while keeping the versions straight

2020-06-26 Thread David Mathog
Thanks for that feedback. Looks like RECORD is the one to use. The names of the directories ending in dist-info seem to be uniformly: package-version.dist_info but the directory names associated with eggs come in a lot of flavors: anndata-0.6.19-py3.6.egg

[Distutils] Re: package management - common storage while keeping the versions straight

2020-06-26 Thread John Thorvald Wodder II
On 2020 Jun 26, at 15:50, David Mathog wrote: > Still, how common is that? Can anybody offer an estimate about what > fraction of packages use different names like that? Scanning through the wheelodex.org database (specifically, a dump from earlier this week) finds 32,517 projects where the

[Distutils] Re: package management - common storage while keeping the versions straight

2020-06-26 Thread John Thorvald Wodder II
(Sending to the list this time.) On 2020 Jun 26, at 15:43, David Mathog wrote: > So by what method could code working outside of python possibly determine that > "yaml" goes with "PyYAML"? By checking all *.dist-info/RECORD files to see which one mentions the "yaml" directory. (top_level.txt

[Distutils] Re: package management - common storage while keeping the versions straight

2020-06-26 Thread David Mathog
On Fri, Jun 26, 2020 at 12:43 PM David Mathog wrote: > So by what method could code working outside of python possibly determine that > "yaml" goes with "PyYAML"? Sorry, I forgot that the information was in PyYAML-5.3.1-py3.6.egg-info/top_level.txt Still, how common is that? Can anybody offer

[Distutils] Re: package management - common storage while keeping the versions straight

2020-06-26 Thread David Mathog
Questions about naming conventions. The vast majority of packages when they install create in site-packages two directories with names like: foobar foobar-1.2.3.dist-info (or egg-info) However PyYAML creates: yaml PyYAML-5.3.1-py3.6.egg-info and there is also this: pkg_resources which is

[Distutils] Re: package management - common storage while keeping the versions straight

2020-06-25 Thread David Mathog
On Thu, Jun 25, 2020 at 12:37 AM Paul Moore wrote: > I think the key message here is that you won't be *re*-inventing the > wheel. This is a wheel that still needs to be invented. It _was_ invented, but it is off round and gives a rough ride. As noted in the first post this: __requires__ =

[Distutils] Re: package management - common storage while keeping the versions straight

2020-06-25 Thread Jason Madden
On Tue, 2020-06-23 at 15:51 -0700, David Mathog wrote: > What I am after is some method of keeping exactly one copy of each > package-version in the common area (ie, one might find foo-1.2, > foo-1.7, and foo-2.3 there), while also presenting only the one > version of each (let's say foo-1.7) to a

[Distutils] Re: package management - common storage while keeping the versions straight

2020-06-25 Thread Paul Moore
On Thu, 25 Jun 2020 at 00:06, David Mathog wrote: > > Thanks for the link. Unfortunately there was not a reference to a > completed package that actually did this. As in, I really do not want > to reinvent the wheel. Ugh, sorry, that's a pun in this context. I think the key message here is

[Distutils] Re: package management - common storage while keeping the versions straight

2020-06-24 Thread Filipe LaĆ­ns
On Tue, 2020-06-23 at 15:51 -0700, David Mathog wrote: > What I am after is some method of keeping exactly one copy of each > package-version in the common area (ie, one might find foo-1.2, > foo-1.7, and foo-2.3 there), while also presenting only the one > version of each (let's say foo-1.7) to a

[Distutils] Re: package management - common storage while keeping the versions straight

2020-06-24 Thread David Mathog
It turned out that the second install was not the cause of the timestamp change in the original. On reviewing "history" it turned out that I had accidentally run the link generation twice. That turned up this (for me) unexpected behavior: mkdir /tmp/foo ls -al /tmp/foo total 16 drwxrwxr-x. 2

[Distutils] Re: package management - common storage while keeping the versions straight

2020-06-24 Thread David Mathog
Thanks for the link. Unfortunately there was not a reference to a completed package that actually did this. As in, I really do not want to reinvent the wheel. Ugh, sorry, that's a pun in this context. Here is a first shot at this, just installing a moderately complicated package in a virtualenv

[Distutils] Re: package management - common storage while keeping the versions straight

2020-06-24 Thread Steve Dower
On 24Jun2020 1923, David Mathog wrote: I think I will experiment a little with pipenv and if necessary after each package install use a script to remove the installed libraries and replace them with a hard link to the one in the common area. Maybe it will be possible to put in those links before

[Distutils] Re: package management - common storage while keeping the versions straight

2020-06-24 Thread David Mathog
On Wed, Jun 24, 2020 at 1:36 AM Thomas Kluyver wrote: > > On Tue, 23 Jun 2020, at 23:51, David Mathog wrote: > > What I am after is some method of keeping exactly one copy of each > > package-version in the common area (ie, one might find foo-1.2, > > foo-1.7, and foo-2.3 there), while also

[Distutils] Re: package management - common storage while keeping the versions straight

2020-06-24 Thread Thomas Kluyver
On Tue, 23 Jun 2020, at 23:51, David Mathog wrote: > What I am after is some method of keeping exactly one copy of each > package-version in the common area (ie, one might find foo-1.2, > foo-1.7, and foo-2.3 there), while also presenting only the one > version of each (let's say foo-1.7) to a

[Distutils] Re: package management - common storage while keeping the versions straight

2020-06-24 Thread Paul Moore
On Wed, 24 Jun 2020 at 00:00, David Mathog wrote: > Does such a beast exist? If so, please point me to it! Basically no, or at least not to my knowledge. The mechanisms exist, in the form of import hooks and similar, to build something like this, but it's not proved to be a common enough