Hi Freddy, Thomas thanks for the well cautioned notes.
I'm vaguely aware that both, nix and conda, as "meta package managers" have a effective way of linking correct sets of packages including the tracing. however both tools never made it into a consistent set of tools that my peers, coworkers, co-maintainers and myself use. as such i want to see the capabilities happen more at the python level. wrt tracing whats from where, as a starting point i'd only packages wheels from pypi (or a custom designated index server) as target for the linking I am well aware that this constraint precludes certain things that nix and conda would do, but the initial isolation from binary dependencies is critical on order to avoid layering that rabbit-hole in at a greater expense. I'd love to see a followup in tooling that would allow to map available/required feature sets from build time dependencies to eventually elevate the issue, but such process will take quite a while because of lack of available contributors One of my initial main goals for a minimal viable solution is to trim down the massive duplication of available source wheels and manylinux wheels more nuanced builds should stay on a classical layout/setup on a per package basis while the details are fleshed out, i certainly would like to see a path towards linking packages build on tools like nix and conda more effectively (in particular removing the need for deep layered and nested symlink trees which i perceived in nix would be fabulous) -- Ronny Am 08.10.18 um 10:20 schrieb Freddy Rietdijk: > What Conda and Nix do is basically having a store with packages, and > then indeed linking them to form environments. While this works, it is > important that one is very careful tracing how the packages are built, > because not doing so can easily cause trouble in case of modules with > extension types. > > On Mon, Oct 8, 2018 at 10:14 AM, Thomas Kluyver <tho...@kluyver.me.uk > <mailto:tho...@kluyver.me.uk>> wrote: > > Have you tried using conda envs? Conda avoids duplication by > unpacking all packages to a shared directory and then hard-linking > them into environments. > > I'm not saying that should preclude doing something similar for > virtualenvs, but it's probably worth looking at existing solutions > before embarking on a change of this magnitude. > > On Mon, Oct 8, 2018, at 8:42 AM, RonnyPfannschmidt wrote: > > hi everyone, > > > > i just did a little spring cleaning, and while i cleaned up my > > ~/Projects folder - the vast amount of that was virtualenvs with > roughly > > the same packages installed - each about 500 mb for like 2-5 mb > of code > > > > this seems excessively wasteful - my impression is that ~98% of the > > files seem to be duplicate. > > > > since the impending eol of python2 and the multi python version > > capabilities of python3 compiled code caches, i'd like to create a > > proposal for a shared folder structure > > > > > > $LIB/python/shared-packages/ should be a root folder in which > unpacked > > wheels would reside > > > > at a initial iteration this solution would suffer from the expanding > > python path issue since each package would have a toplevel > sys.path entry. > > > > on order to later on speed this up a simple api can be provided that > > makes python aware of the metatadata locations and the map > > toplevel/namespace names directly to import locations > > > > > > for example like > > > > shared-packages.pth > > > > import sys;sys.path.insert(0, > > "$LIB/python/shared-packages/shared_packages_support-1.0-py3.whl/"); > > import > > > shared_packages_support;sys.path.pop(0);shared_packages_support.enable() > > > > shared_packages.json > > > > { > > > > "pytest": "pytest-3.8.8-py2.py3.whl"; > > > > } > > > > > > in the aftermath, when shared packages support and fast > import/metadata > > for shared packages where implemented, the sizes of most virtualenvs > > would shrink significantly > > > > > > -- Ronny > > -- > > Distutils-SIG mailing list -- distutils-sig@python.org > <mailto:distutils-sig@python.org> > > To unsubscribe send an email to distutils-sig-le...@python.org > <mailto:distutils-sig-le...@python.org> > > > https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ > <https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/> > > Message archived at > > > > https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/JWX7VFJRLWVIIBP4NZ5MEPRVCYTV4ZJT/ > > <https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/JWX7VFJRLWVIIBP4NZ5MEPRVCYTV4ZJT/> > -- > Distutils-SIG mailing list -- distutils-sig@python.org > <mailto:distutils-sig@python.org> > To unsubscribe send an email to distutils-sig-le...@python.org > <mailto:distutils-sig-le...@python.org> > https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ > <https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/> > Message archived at > > https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/XAZPBTKGEWVDLE3PJITY4JQ3YGK3NL7G/ > > <https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/XAZPBTKGEWVDLE3PJITY4JQ3YGK3NL7G/> > > > > -- > Distutils-SIG mailing list -- distutils-sig@python.org > To unsubscribe send an email to distutils-sig-le...@python.org > https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ > Message archived at > https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/3L24OT5FOHPBNTQHJYRNKR4HHDQLRD4X/
-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/TYUKA44NU3VLS7X4RQOZXRUILU25RZ5Q/
