Hi Freddy, im well aware what nix currently does for python packages and suffered my fair share from it.
What i want to do is simply store those wheels that pip would first generate and unpack into each environment into a location where each environment shares the unpacked files more directly im not going to expand uppon my perceived shortcomings of nix as i know ,it since its irrelevant to this discussion and not something i have the time and motivation to fix. as far as impurities go, the behaviour i aim for would be moslty like virtualenv but without the file duplication. I beleive nix could also benefit from parts of such a mechanism. -- Ronny Am Montag, den 30.10.2017, 20:35 +0100 schrieb Freddy Rietdijk: > Hi Ronny, > > What you describe here is, as you know, basically what the Nix > package manager does. You could create something similar specifically > for Python, like e.g. `ied` is for Node [2], or Spack, which is > written in Python. But then how are you going to deal with other > system libraries, and impurities? And you will have to deal with > them, because depending on how you configure Python packages that > depend on them (say a `numpy`), their output will be different. Or > would you choose to ignore this? > > Freddy > > [1] https://nixos.org/nix/ > [2] https://github.com/alexanderGugel/ied > [3] https://spack.io/ > > On Mon, Oct 30, 2017 at 8:16 PM, RonnyPfannschmidt <opensource@ronnyp > fannschmidt.de> wrote: > > Hi everyone, > > > > since a while now various details of installing python packages in > > virtualenvs caused me grief > > > > a) typically each tox folder in a project is massive, and has a lot > > of > > duplicate files, recreating them, managing and iterating them takes > > quite a while > > b) for nicely separated deployments, each virtualenv for an > > application > > takes a few hundred megabytes - that quickly can saturate disk > > space > > even if a reasonable amount was reserved > > c) installation and recreation of virtualenvs with the same set of > > packages takes quite a while (even with pip caches this is slow, > > and > > there is no good reason to avoid making it completely > > instantaneous) > > > > in order to elevate those issues i would like to propose a new > > installation layout, > > where instead of storing each distribution in every python all > > distributions would share a storage, and each individual > > environment > > would only have references to the packages that where > > "installed/activated" for them > > > > this would massively reduce time required to create the contents of > > the > > environments and also the space required > > > > since blindly expanding sys.path would lead to similar performance > > issues as where seen with setuptools/buildout multi-version > > installs, > > this mechanism would also need a element on sys.meta_path that > > handles > > inexpensive dispatch to the toplevels and metadata files of each > > packages (off hand i would assume linear walking of hundreds of > > entries > > simply isn't that effective) > > > > however there would be need for some experimentation to see what > > tradeoff is sensible there > > > > I hope this mail will spark enough discussion to enable the > > creation of > > a PEP and a prototype. > > > > > > Best, Ronny > > > > > > > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG@python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > > _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig