, the behaviour i aim for would be moslty like virtualenv but without the file duplication.
For what it’s worth, conda environments use hard links where possible, so limiting duplication... Maybe conda would solve your problem... -CHB I beleive nix could also benefit from parts of such a mechanism. -- Ronny Am Montag, den 30.10.2017, 20:35 +0100 schrieb Freddy Rietdijk: Hi Ronny, What you describe here is, as you know, basically what the Nix package manager does. You could create something similar specifically for Python, like e.g. `ied` is for Node [2], or Spack, which is written in Python. But then how are you going to deal with other system libraries, and impurities? And you will have to deal with them, because depending on how you configure Python packages that depend on them (say a `numpy`), their output will be different. Or would you choose to ignore this? Freddy [1] https://nixos.org/nix/ [2] https://github.com/alexanderGugel/ied [3] https://spack.io/ On Mon, Oct 30, 2017 at 8:16 PM, RonnyPfannschmidt <opensource@ronnyp fannschmidt.de> wrote: Hi everyone, since a while now various details of installing python packages in virtualenvs caused me grief a) typically each tox folder in a project is massive, and has a lot of duplicate files, recreating them, managing and iterating them takes quite a while b) for nicely separated deployments, each virtualenv for an application takes a few hundred megabytes - that quickly can saturate disk space even if a reasonable amount was reserved c) installation and recreation of virtualenvs with the same set of packages takes quite a while (even with pip caches this is slow, and there is no good reason to avoid making it completely instantaneous) in order to elevate those issues i would like to propose a new installation layout, where instead of storing each distribution in every python all distributions would share a storage, and each individual environment would only have references to the packages that where "installed/activated" for them this would massively reduce time required to create the contents of the environments and also the space required since blindly expanding sys.path would lead to similar performance issues as where seen with setuptools/buildout multi-version installs, this mechanism would also need a element on sys.meta_path that handles inexpensive dispatch to the toplevels and metadata files of each packages (off hand i would assume linear walking of hundreds of entries simply isn't that effective) however there would be need for some experimentation to see what tradeoff is sensible there I hope this mail will spark enough discussion to enable the creation of a PEP and a prototype. Best, Ronny _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
