What Conda and Nix do is basically having a store with packages, and then
indeed linking them to form environments. While this works, it is important
that one is very careful tracing how the packages are built, because not
doing so can easily cause trouble in case of modules with extension types.
On Mon, Oct 8, 2018 at 10:14 AM, Thomas Kluyver <tho...@kluyver.me.uk>
wrote:
> Have you tried using conda envs? Conda avoids duplication by unpacking all
> packages to a shared directory and then hard-linking them into environments.
>
> I'm not saying that should preclude doing something similar for
> virtualenvs, but it's probably worth looking at existing solutions before
> embarking on a change of this magnitude.
>
> On Mon, Oct 8, 2018, at 8:42 AM, RonnyPfannschmidt wrote:
> > hi everyone,
> >
> > i just did a little spring cleaning, and while i cleaned up my
> > ~/Projects folder - the vast amount of that was virtualenvs with roughly
> > the same packages installed - each about 500 mb for like 2-5 mb of code
> >
> > this seems excessively wasteful - my impression is that ~98% of the
> > files seem to be duplicate.
> >
> > since the impending eol of python2 and the multi python version
> > capabilities of python3 compiled code caches, i'd like to create a
> > proposal for a shared folder structure
> >
> >
> > $LIB/python/shared-packages/ should be a root folder in which unpacked
> > wheels would reside
> >
> > at a initial iteration this solution would suffer from the expanding
> > python path issue since each package would have a toplevel sys.path
> entry.
> >
> > on order to later on speed this up a simple api can be provided that
> > makes python aware of the metatadata locations and the map
> > toplevel/namespace names directly to import locations
> >
> >
> > for example like
> >
> > shared-packages.pth
> >
> > import sys;sys.path.insert(0,
> > "$LIB/python/shared-packages/shared_packages_support-1.0-py3.whl/");
> > import
> > shared_packages_support;sys.path.pop(0);shared_packages_support.enable()
> >
> > shared_packages.json
> >
> > {
> >
> > "pytest": "pytest-3.8.8-py2.py3.whl";
> >
> > }
> >
> >
> > in the aftermath, when shared packages support and fast import/metadata
> > for shared packages where implemented, the sizes of most virtualenvs
> > would shrink significantly
> >
> >
> > -- Ronny
> > --
> > Distutils-SIG mailing list -- distutils-sig@python.org
> > To unsubscribe send an email to distutils-sig-le...@python.org
> > https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> > Message archived at
> > https://mail.python.org/mm3/archives/list/distutils-sig@
> python.org/message/JWX7VFJRLWVIIBP4NZ5MEPRVCYTV4ZJT/
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> Message archived at https://mail.python.org/mm3/
> archives/list/distutils-sig@python.org/message/
> XAZPBTKGEWVDLE3PJITY4JQ3YGK3NL7G/
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/3L24OT5FOHPBNTQHJYRNKR4HHDQLRD4X/
