Re: [Nix-dev] Patch for pythonPackages.numpy
Excerpts from Florian Friesdorf's message of Mon Jun 18 18:14:48 +0200 2012: pth processing is not the issue, the issue is that some packages (like matplotlib) do not create them as part of their install. I don't know. I just ran a a simple gtk sample - and it worked (after making sure gtk is recognized when building matplotlib). The main difference is that I looked up what PYTHONUSERBASE does and did the same for multiple paths. I didn't even try to understand all details. I never had real trouble due to that - but it looks like the the traditional way in nixpkgs caused issues (eg virtinst). If you know why I'd like to learn about it so that I stop making false assumptions .. :) Some additional small changes I did are always adding $out to NIX_PYTHON_SITES when python is added as buildinput - so things always work without libraries telling you trying to install into location not read by PYTHONPATH or the like. Maybe its overkill - it stops bothering me that way. How do you deal with that? You write below that your patch adds a pth file to solve problems with pygtk. The pygtk comments are outdated. I assumed that import pygtk; pygtk.require(2.0) is still the way to go but its no longer the case. Everyone is using 2.0 now - so people just drop that require line usually. Instead of symlinking everything I just added one .pth file. I'm don't have a strong feeling about which is better. And that .pth file is added whenever a gtk-2.0 dir is found by using a small setup-hook snippet so that py-gnome and the like just work as well (and whatever package is using the same directory layout) Marc Weber ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Patch for pythonPackages.numpy
Florian Friesdorf writes: I don't think this is related in any way at all to your patch or to the python branch, but it's pretty effective at preventing me from building numpy. The python branch is based on a rather recent trunk. You could try to apply the patch to nixpkgs from the nixos-unstable channel. I just tried, with the current nixos-unstable version (nixpkgs-1.0pre19835_1c5f4b8). Same problem. I'll probably have to go further back in time. Does anyone happen to know a sufficiently recent version from which one can at least build stdenv under MacOS X? Konrad. ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Patch for pythonPackages.numpy
Florian Friesdorf writes: Could you try whether the attached patch also solves building on OS X (Konrad) and whether matplotlib builds for you (Peter)? Since I recently upgraded to MacOS X 10.7, I decided to reinstall Nix (1.0) and Nixpkgs from scratch to see how it works out. I downloaded the python branch you mentioned and added your patch for numpy. Then I typed nix-build -A pythonPackages.numpy ./pkgs/top-level/all-packages.nix This leads to a failure in building libsigsegv-2.10 (due to a failing test), which is a dependency of gawk-4.0.0, which in turn is a dependency of stdenv. libsigsegv 2.10 is supposed to work fine under MacOSX 10.7, so I suppose the failure is due to the build environment in nixpkgs. I don't think this is related in any way at all to your patch or to the python branch, but it's pretty effective at preventing me from building numpy. Maybe the ATLAS stuff in the patch is guilty? Well possible. Getting ATLAS right is not a trivial job. Konrad. ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Patch for pythonPackages.numpy
Hi Marc,
On Sun, 17 Jun 2012 17:16:21 +0200, Marc Weber marco-owe...@gmx.de wrote:
I'm against that patch because it heavy monkey patching does not solving the
problem at the root of the evil. My old NIX_PYTHON_SITES patch does.
Now there is evidence that such a patch is required just to make things
work without additional work.
What does it do? Basically it patches python import code introducing
NIX_PYTHON_SITES which behaves as PYTHON_USER_BASE *but* allows multiple
paths to be passed.
(..)
What might be the difference? My code definitely picks up .pth files.
If you use PYTHONPATH only it may not happen. (I haven't reverified it)
pth processing is not the issue, the issue is that some packages (like
matplotlib) do not create them as part of their install.
How do you deal with that? You write below that your patch adds a pth
file to solve problems with pygtk. This sounds very similar to the
approach we take:
- PYTHONPATH is used to set sys.path
- pth files are processed as normal in exec prefixes
- where this is not sufficient, include customized site.py that
processes pth files in all dirs of PYTHONPATH
- use propagatedBuildInputs (in contrast to buildInputs) if a package
does not use setuptools and therefor fails to create pth files and
therefor lacks references of other packages hashes.
About pygtk: my patch adds a .pth file instead of creating symlinks.
Should have the same effect.
The pygtk.require('2.0') stuff (still found on pygtk documentation)
seems to be no longer used because nobody is using gtk-1.0 anymore.
regards
florian
--
Florian Friesdorf f...@chaoflow.net
GPG FPR: 7A13 5EEE 1421 9FC2 108D BAAF 38F8 99A3 0C45 F083
Jabber/XMPP: f...@chaoflow.net
IRC: chaoflow on freenode,ircnet,blafasel,OFTC
pgpSBkRGUWF2g.pgp
Description: PGP signature
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Patch for pythonPackages.numpy
On Mon, 18 Jun 2012 17:15:20 +0200, Konrad Hinsen konrad.hin...@fastmail.net wrote: Florian Friesdorf writes: Could you try whether the attached patch also solves building on OS X (Konrad) and whether matplotlib builds for you (Peter)? Since I recently upgraded to MacOS X 10.7, I decided to reinstall Nix (1.0) and Nixpkgs from scratch to see how it works out. I downloaded the python branch you mentioned and added your patch for numpy. Then I typed nix-build -A pythonPackages.numpy ./pkgs/top-level/all-packages.nix This leads to a failure in building libsigsegv-2.10 (due to a failing test), which is a dependency of gawk-4.0.0, which in turn is a dependency of stdenv. libsigsegv 2.10 is supposed to work fine under MacOSX 10.7, so I suppose the failure is due to the build environment in nixpkgs. I don't think this is related in any way at all to your patch or to the python branch, but it's pretty effective at preventing me from building numpy. The python branch is based on a rather recent trunk. You could try to apply the patch to nixpkgs from the nixos-unstable channel. -- Florian Friesdorf f...@chaoflow.net GPG FPR: 7A13 5EEE 1421 9FC2 108D BAAF 38F8 99A3 0C45 F083 Jabber/XMPP: f...@chaoflow.net IRC: chaoflow on freenode,ircnet,blafasel,OFTC pgphnqrWyD2Oe.pgp Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Patch for pythonPackages.numpy
Hi Florian, Could you try whether the attached patch also solves building on OS X (Konrad) and whether matplotlib builds for you (Peter)? matplotlib compiles fine with that patch applied. :-) Take care, Peter ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Patch for pythonPackages.numpy
I'm against that patch because it heavy monkey patching does not solving the
problem at the root of the evil. My old NIX_PYTHON_SITES patch does.
Now there is evidence that such a patch is required just to make things
work without additional work.
What does it do? Basically it patches python import code introducing
NIX_PYTHON_SITES which behaves as PYTHON_USER_BASE *but* allows multiple
paths to be passed.
Why isn't my patch in nixpkgs? I tried getting it in but had a rebase
bug that time and due to my job I wasn't able to fix it within 3 days.
So Eelco Dolstra has had no chance but reverting my patch that time -
because those people being active on python stuff didn't believe in its
necessity.
As always there are multiple ways to solve a trouble - its only me
thinking that my solution is the simpler one in the long run because it
solves the trouble at the root instead of monkey patching derivations
later.
You can find the code at:
https://github.com/MarcWeber/nixpkgs
branch experimental/nix-python-sites
This makes all work:
- matplotlib (including gtk gui by passing pygtk)
- virtinst python bindings
- virtmanager (which previously failed finding libraries and failed due
to virtinst issues)
Note: I haven't tested virtmanager yet, but I consider the window
showing up a success.
What's left? Rewrite existing code to use NIX_PYTHON_SITES such as wicd.
What might be the difference? My code definitely picks up .pth files.
If you use PYTHONPATH only it may not happen. (I haven't reverified it)
If you like it discuss or pick the patch - you can create a patch file
by exporting the top-git branch easily.
Should the env var contain the python version ? eg NIX_PYTHON27_SITES ?
Don't know - python folks decided against it (forgot about the source).
I'd say keep it simple unless required otherwise.
About pygtk: my patch adds a .pth file instead of creating symlinks.
Should have the same effect.
The pygtk.require('2.0') stuff (still found on pygtk documentation)
seems to be no longer used because nobody is using gtk-1.0 anymore.
Marc Weber
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Patch for pythonPackages.numpy
Hi Marc, On Sun, 17 Jun 2012 17:16:21 +0200, Marc Weber marco-owe...@gmx.de wrote: I'm against that patch because it heavy monkey patching does not solving the problem at the root of the evil. My old NIX_PYTHON_SITES patch does. I don't understand how default include and lib dirs relate to NIX_PYTHON_SITES. Are you referring to the patch introducing create_pth_for_propagated_inputs? regards florian -- Florian Friesdorf f...@chaoflow.net GPG FPR: 7A13 5EEE 1421 9FC2 108D BAAF 38F8 99A3 0C45 F083 Jabber/XMPP: f...@chaoflow.net IRC: chaoflow on freenode,ircnet,blafasel,OFTC pgpFYv1BcYYGl.pgp Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Patch for pythonPackages.numpy
Hi Konrad, hi Peter,
thank you for your patch and sorry for the late response, too busy with
other things. Should get better now.
Could you try whether the attached patch also solves building on OS X
(Konrad) and whether matplotlib builds for you (Peter)?
more comments inline:
On Mon, 30 Apr 2012 16:24:18 +0200, Konrad Hinsen konrad.hin...@fastmail.net
wrote:
Peter Simons writes:
Building NumPy on MacOS fails with the current state of nixpkgs, and I
suspect it can fail on other platforms as well, as the problems I found
are not specific to MacOS. With the attached patch, it builds correctly,
at least under MacOS.
I applied your patch in r33941, but that change caused build failures
for matplotlib on Linux (and probably on other platforms as well which
Hydra doesn't automatically check):
http://hydra.nixos.org/build/2490567
Do you by chance know how to fix these problems?
The error message You must install numpy 1.1 or later to build can
only have one cause: an import numpy failed during installation of
matplotlib. And that means that the numpy in your Nix store is not on
PYTHONPATH when matplotlib is built. I haven't yet studied how Nix
handles Python packages, so I'd better shut up now. Meaning that I
don't have a solution at this time.
We are reworking python dependency handling in the python branch[1].
Nix figures out runtime dependencies by scanning the installed files of
a package for hashes of other packages. This works very well with
ELF files as they contain the path to their dependencies.
It also works with pth files, which python (or at least
setuptools/distribute) uses to add dependencies to PYTHONPATH. In the
python branch this works nicely with all packages using setuptools.
However, matplotlib does things differently and does not contain
references to its python dependencies.
Solution path:
- all dependencies are handled via pth files
- pth files are explicitly created for all python dependencies that are
added to propagatedBuildInputs (in contrast to buildInputs).
- dependencies that fail to show up as runtime dependencies by natural
means are added to propagatedBuildInputs
There is work in progress to create such pth files[2]. I hacked the
create_pth_for_propagated_inputs, it did not work, and I had to move
focus to more urgent matters.
Any help is greatly appreciated.
Note that matplotlib fails to build under MacOS because some of its
dependencies fail, so I can't analyze the problem myself for now.
Are you sure that matplotlib builds correctly without my patch for
NumPy? It doesn't affect the accessibility of NumPy at all, so I don't
see what could be the problem. It could potentially cause a build
failure for NumPy, but not for matplotlib.
Maybe the ATLAS stuff in the patch is guilty?
With the attached modified patch (applied to the python branch) numpy
and matplotlib compile fine. Please check whether it solves your problem
and I will commit it.
BTW, my NumPy patch mainly removes the default directories for library
searches. Matplotlib needs a similar patch as well, the evidence being
the line
basedirlist is: ['/usr/local', '/usr']
I'll look into that when I get the dependencies to build.
Great!
regards
florian
[1] https://github.com/chaoflow/nixpkgs
[2]
https://github.com/chaoflow/nixpkgs/blob/python/pkgs/development/python-modules/generic/wrap.sh#L65
From 60bb40c3c01e5e2d95db28598dd879273d50df62 Mon Sep 17 00:00:00 2001
From: Florian Friesdorf f...@chaoflow.net
Date: Sun, 17 Jun 2012 00:49:46 +0200
Subject: [PATCH] numpy: empty default paths - modified patch from Konrad
Hinsen
---
pkgs/top-level/python-packages.nix | 11 +++
1 file changed, 11 insertions(+)
diff --git a/pkgs/top-level/python-packages.nix b/pkgs/top-level/python-packages.nix
index 43d8dee..00e07d5 100644
--- a/pkgs/top-level/python-packages.nix
+++ b/pkgs/top-level/python-packages.nix
@@ -1370,6 +1370,17 @@ let pythonPackages = python.modules // rec {
sha256 = 1pawfmf7j7pd3mjzhmmw9hkglc2qdirrkvv29m5nsmpf2b3ip2vq;
};
+patchPhase =
+ ''
+substituteInPlace ./numpy/distutils/system_info.py \
+ --replace default_lib_dirs = filter(os.path.isdir, default_lib_dirs) \
+default_lib_dirs = [] \
+ --replace default_include_dirs = filter(os.path.isdir, default_include_dirs) \
+default_include_dirs = [] \
+ --replace default_src_dirs = filter(os.path.isdir, default_src_dirs) \
+default_src_dirs = ['.']
+ '';
+
# TODO: add ATLAS=${pkgs.atlas}
installCommand = ''
export BLAS=${pkgs.blas} LAPACK=${pkgs.liblapack}
--
1.7.10.4
--
Florian Friesdorf f...@chaoflow.net
GPG FPR: 7A13 5EEE 1421 9FC2 108D BAAF 38F8 99A3 0C45 F083
Jabber/XMPP: f...@chaoflow.net
IRC: chaoflow on freenode,ircnet,blafasel,OFTC
pgpCryT1oQkQX.pgp
Description: PGP signature
Re: [Nix-dev] Patch for pythonPackages.numpy
Peter Simons writes: Building NumPy on MacOS fails with the current state of nixpkgs, and I suspect it can fail on other platforms as well, as the problems I found are not specific to MacOS. With the attached patch, it builds correctly, at least under MacOS. I applied your patch in r33941, but that change caused build failures for matplotlib on Linux (and probably on other platforms as well which Hydra doesn't automatically check): http://hydra.nixos.org/build/2490567 Do you by chance know how to fix these problems? The error message You must install numpy 1.1 or later to build can only have one cause: an import numpy failed during installation of matplotlib. And that means that the numpy in your Nix store is not on PYTHONPATH when matplotlib is built. I haven't yet studied how Nix handles Python packages, so I'd better shut up now. Meaning that I don't have a solution at this time. Note that matplotlib fails to build under MacOS because some of its dependencies fail, so I can't analyze the problem myself for now. Are you sure that matplotlib builds correctly without my patch for NumPy? It doesn't affect the accessibility of NumPy at all, so I don't see what could be the problem. It could potentially cause a build failure for NumPy, but not for matplotlib. BTW, my NumPy patch mainly removes the default directories for library searches. Matplotlib needs a similar patch as well, the evidence being the line basedirlist is: ['/usr/local', '/usr'] I'll look into that when I get the dependencies to build. Konrad. ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Patch for pythonPackages.numpy
Hi Konrad, Building NumPy on MacOS fails with the current state of nixpkgs, and I suspect it can fail on other platforms as well, as the problems I found are not specific to MacOS. With the attached patch, it builds correctly, at least under MacOS. I applied your patch in r33941, but that change caused build failures for matplotlib on Linux (and probably on other platforms as well which Hydra doesn't automatically check): http://hydra.nixos.org/build/2490567 Do you by chance know how to fix these problems? If in doubt, I'd rather revert that patch again, because it doesn't seem to improve matters. Take care, Peter ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
