I'm +1 on this because I encountered lots of problems with PYTHONPATH, especially for programs that have their own module loading logic (e.g. gunicorn, some Django code).
I don't know what new problems this change will introduce (other than what you mentioned) but since this is closer to how virtualenv works I suspect it'll be manageable. ~ On 1 November 2016 at 10:22, Freddy Rietdijk <[email protected]> wrote: > Hi, > > Currently we use PYTHONPATH > <https://docs.python.org/2/using/cmdline.html#envvar-PYTHONPATH> a lot in > Nixpkgs to let applications and the interpreter find Python modules. This > typically works fine, but there are problems with this method and so I > would like to get rid of it and use only `python.withPackages` which uses > `python.buildEnv`. > > The two main issues with the use of PYTHONPATH: > > 1. Before we used `--prefix $PYTHONPATH`, but this would leak > PYTHONPATH into subprocesses, which is especially problematic when both > parent and child depend on Python but of different versions. `--set > $PYTHONPATH` would solve that issue, but that makes it impossible to add > other (impure) paths, which is an important feature. This also breaks > alternative shells like `ipython`. This issue is currently solved by > extending `sys.path` in the Python applications. > 2. Limits the use of multiple outputs. When moving a module of a > package into a separate output it becomes problematic to load this again, > since just adding the module to PYTHONPATH typically doesn't work because > the order matters. While Python modules are typically small, and build > fast, rebuilding can take a lot of time in cases like `matplotlib` which > supports multiple backends and is depended on by quite some packages. > > A method that is more reliable is building an environment with symbolic > links to all the modules, and wrapping the applications with a wrapper that > sets PYTHONHOME > <https://docs.python.org/2/using/cmdline.html#envvar-PYTHONHOME>. This is > exactly what `python.buildEnv` does, and it solves both 1) and 2). > > `PYTHONPATH` is mainly constructed with the Python interpreter setupHook. > It is used in `buildPythonPackage` for building the package, and after > installing it is extended so the tests can run. Furthermore, `PYTHONPATH` > is set by the `setupHook` when using `nix-shell` like `nix-shell -p > pythonPackages.numpy`. > > I think we can get rid of the setupHook. For the building we can create an > env. This would be able to support multiple outputs as inputs, but will be > more expensive than setting PYTHONPATH. For the tests we do add the newly > constructed package to PYTHONPATH; there's no way around it and it doesn't > cause any problems either. > > The biggest impact will be on how `nix-shell` is used. It won't be > possible anymore to use it as shown before, instead `nix-shell -p > 'python.withPackages(ps:[ps.numpy])'` would have to be used. While it is > possible to keep the setupHook (or use it as a shellHook) it will be > unreliable in the case of multiple outputs. > > What do you think about this change? Do you see any problems with it? Any > other suggestions? > > Freddy > > _______________________________________________ > nix-dev mailing list > [email protected] > http://lists.science.uu.nl/mailman/listinfo/nix-dev > >
_______________________________________________ nix-dev mailing list [email protected] http://lists.science.uu.nl/mailman/listinfo/nix-dev
