Hi Ronny, What you describe here is, as you know, basically what the Nix package manager does. You could create something similar specifically for Python, like e.g. `ied` is for Node [2], or Spack, which is written in Python. But then how are you going to deal with other system libraries, and impurities? And you will have to deal with them, because depending on how you configure Python packages that depend on them (say a `numpy`), their output will be different. Or would you choose to ignore this?
Freddy [1] https://nixos.org/nix/ [2] https://github.com/alexanderGugel/ied [3] https://spack.io/ On Mon, Oct 30, 2017 at 8:16 PM, RonnyPfannschmidt < opensou...@ronnypfannschmidt.de> wrote: > Hi everyone, > > since a while now various details of installing python packages in > virtualenvs caused me grief > > a) typically each tox folder in a project is massive, and has a lot of > duplicate files, recreating them, managing and iterating them takes > quite a while > b) for nicely separated deployments, each virtualenv for an application > takes a few hundred megabytes - that quickly can saturate disk space > even if a reasonable amount was reserved > c) installation and recreation of virtualenvs with the same set of > packages takes quite a while (even with pip caches this is slow, and > there is no good reason to avoid making it completely instantaneous) > > in order to elevate those issues i would like to propose a new > installation layout, > where instead of storing each distribution in every python all > distributions would share a storage, and each individual environment > would only have references to the packages that where > "installed/activated" for them > > this would massively reduce time required to create the contents of the > environments and also the space required > > since blindly expanding sys.path would lead to similar performance > issues as where seen with setuptools/buildout multi-version installs, > this mechanism would also need a element on sys.meta_path that handles > inexpensive dispatch to the toplevels and metadata files of each > packages (off hand i would assume linear walking of hundreds of entries > simply isn't that effective) > > however there would be need for some experimentation to see what > tradeoff is sensible there > > I hope this mail will spark enough discussion to enable the creation of > a PEP and a prototype. > > > Best, Ronny > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig >
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
