Re: [Nix-dev] Zoom package?
Hi, I needed it a while ago and packaged it: https://github.com/aherrmann/zoom_client_nix It was a one time use for me at the time and it only worked on stable and was broken on master (see Readme), so I didn't go further to try and add it to nixpkgs. Best, Andreas On Friday 07 July 2017 10:41:54 Mark Gardner via nix-dev wrote: > Has anyone started packaging Zoom (https://zoom.us/) for NixOS? > > Mark > ___ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] use fetchgitPrivate with NixOps (permission denied)
Hi Micheal, On Sunday 02 April 2017 20:33:59 m...@thought2.de wrote: > nixops deploy -d network > > [..] > Initialized empty Git repository in /nix/store/hn...d4c3/.git/ > Can't open user config file /home/me/.ssh/config: Permission denied > fatal: Could not read from remote repository. > > > Anybody ideas how this could be fixed? Do you have `NIX_REMOTE=daemon` in your environment? That would be the case if you're on NixOS or a multi-user Nix installation. > For using fetchgitPrivate one should put a ssh config file and an auth > socket on the NIX_PATH: > > NIX_PATH=$NIX_PATH:ssh-config-file=/home/me/.ssh/config:ssh-auth-sock=$SSH_AUTH_SOCK > > ls -la /home/me/.ssh/config > > -rw-r--r-- 1 me users 108 2. Apr 19:37 /home/m/.ssh/config > > echo $SSH_AUTH_SOCK > > /run/user/1000/ssh-agent > > So the file should have the right permissions. If you're using the Nix daemon, then the build job will be executed by a different user. That user will need read permissions on the ssh config. What does `ls -ld /home/me/.ssh` say? If it has permissions `drwx--` then the Nix build user will not be able to open that directory and read the ssh config. I personally don't like the idea of giving broad permissions on my ssh config. To enable `fetchgitPrivate` I have a separate ssh config in `~/.nixpkgs/ssh` which is world readable. In that directory I have a dedicated key without pass phrase for Nix build users which is enabled on GitHub and the like as a deployment key. That way the worst somebody who steals the private key from there can do is to read the repository, which at least on the machine I'm working on, they could do anyway because the code ends up in /nix/store which is world readable. Since the key has no pass phrase I also don't need an auth socket. Hope that helps. Best, Andreas ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-commits] [NixOS/nixpkgs] 481630: blitz: switch back to boost-1.60
Branch: refs/heads/release-17.03 Home: https://github.com/NixOS/nixpkgs Commit: 481630a5f621a8b213a6a017ef3a16fdda03da2f https://github.com/NixOS/nixpkgs/commit/481630a5f621a8b213a6a017ef3a16fdda03da2f Author: Andreas Herrmann <andreas...@gmx.ch> Date: 2017-03-02 (Thu, 02 Mar 2017) Changed paths: M pkgs/top-level/all-packages.nix Log Message: --- blitz: switch back to boost-1.60 The build fails with boost-1.62. More specifically, the test of the boost-serialization integration fails due to the protected destructor in the class template `MemoryBlock`. (cherry picked from commit 5a68d5484e9a3c654ac768093ca29026302a7c23) ___ nix-commits mailing list nix-comm...@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-commits
[Nix-commits] [NixOS/nixpkgs] 5a68d5: blitz: switch back to boost-1.60
Branch: refs/heads/master Home: https://github.com/NixOS/nixpkgs Commit: 5a68d5484e9a3c654ac768093ca29026302a7c23 https://github.com/NixOS/nixpkgs/commit/5a68d5484e9a3c654ac768093ca29026302a7c23 Author: Andreas Herrmann <andreas...@gmx.ch> Date: 2017-03-02 (Thu, 02 Mar 2017) Changed paths: M pkgs/top-level/all-packages.nix Log Message: --- blitz: switch back to boost-1.60 The build fails with boost-1.62. More specifically, the test of the boost-serialization integration fails due to the protected destructor in the class template `MemoryBlock`. ___ nix-commits mailing list nix-comm...@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-commits
Re: [Nix-dev] nix-shell pro/con (was: Python 3 as default)
On Wednesday 15 February 2017 22:08:26 Profpatsch wrote: > On 17-02-15 09:10pm, Ricardo M. Correia wrote: > > Personally, I try to avoid using "nix-shell" as much as possible because I > > want "nixos-rebuild switch" to atomically install (and upgrade) all the > > software that I usually need, including development packages. > > > > With nix-shell, I risk having the packages I need become outdated (without > > me noticing) and/or become garbage collected (which I do every day > > automatically) and unavailable if I lose Internet access (e.g. because I'm > > traveling). > > packages being not cached is a largely orthogonal problem; > for example it would be quite easy to create a script that fetches > the nix-shell deps for all your projects and creates a folder with > gc-roots. I've written such a script a while ago: https://gist.github.com/aherrmann/51b56283f9ed5853747908fbab907316 It places the gcroots in separate folders under `~/.nix-shell`. If you run it as `pnix-shell -c` it will reevaluate the Nix expression. This way you can enforce updates. ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Introducing Nixpkgs Overlays
I am using `overridePackages` for a similar purpose already.
(Simplified for ease of presentation.)
One top-level file ties the overlays together
and serves as a replacement for ``.
`mynixpkgs.nix`:
``` nix
{ machine }:
let
pkgs = import { };
machineOverlay =
{ machineA = import ./machineA.nix;
machineB = import ./machineB.nix;
}.${machine} or (throw "unknown machine: ${machine}!")
;
extends' =
lhs: rhs: self: super:
let super' = rhs self super; in
super' // lhs self (super // super');
inherit (pkgs.lib) foldl' flip;
in
pkgs.overridePackages (
foldl' (flip extends') (_:_:{}) [
(import ./extraPkgs.nix)
machineOverlay
# ...
]
)
```
I can add my own private packages in their own overlay layer.
`extraPkgs.nix`:
``` nix
self: super: with self; {
foo = callPackage ./foo { };
bar = callPackage ./bar { };
}
```
And I can add machine specific settings
in another overlay layer.
`machineA.nix`:
``` nix
self: super: with self; {
boost = super.boost.override { mpi = openmpi; };
fftw = super.fftw.overrideDerivation (attrs: {
configureFlags = attrs.configureFlags ++ [ "--enable-avx" ];
});
foo = (super.foo.override {
# Some machine specific parameters
}).overrideDerivation (attrs: {
# Some machine specific attributes
});
}
```
`machineB.nix`:
``` nix
self: super: with self; {
# Different machine specific settings.
}
```
Having such an overlay mechanism built into nixpkgs
would simplify this pattern.
I would welcome such a change.
Best, Andreas
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] truly persistent nix-shell
On Sunday 20 November 2016 15:00:55 Ruben Astudillo wrote: > [...] they > provide the following answer [3] > > $ cd > $ mkdir gcroots > $ nix-shell ./shell.nix --pure --indirect --add-root gcroots/dep > $ exit > $ nix-collect-garbage > $ nix-shell ./shell.nix > > [3]: https://github.com/NixOS/nixpkgs/issues/5623#issuecomment-74048047 Thank you for pointing me there. I never knew that `nix-shell` accepts these flags. You actually don't even need to create the `gcroots` directory before. `nix-shell` will create it if necessary > So this maybe makes you command redundant (although less obsucure than > this commands). Maybe is worthwhile document this behaviour & simplify > the flags to nix-shell to obtain this result? Documenting it would probably already go a long way. Is there any reason why it is not mentioned in the man-page, yet? Best, Andreas ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] truly persistent nix-shell
Hi, I've written a small shell function that creates truly persistent nix-shells (see below). [1] I thought it might be of use to others. Let me know what you think of it. What do I mean by "truly persistent"? The Nix Wiki entry about development environments has a section about "persistent nix-shell environments". [2] It says: > nix-shell evaluates an expression and drops you into the environment in one > command. > This is perfect for temporary development environments, > but you may also want to create a persistent environment > which you can load again at a later date without re-evaluating the expression. It then goes on to explain how to achieve that: > $ nix-instantiate . --indirect --add-root $PWD/shell.drv > $ nix-shell $PWD/shell.drv However, the above comes short of what I, IMHO, would expect from a _persistent_ nix-shell. In particular, the above will not protect the dependencies of the nix-shell from garbage collection, i.e. the packages you want to use within the shell. Imagine you have configured a nix-shell with a lot of overriden packages that take a long time to build. If you then create a "persistent" nix-shell according to the above description. And if you then run the garbage collector. Then all these packages will be collected (unless another gc-root depends on them) and you will have to build them once again. It is possible to avoid this issue by creating a gc-root for every package that the nix-shell pulls in: $ nix-store -r $(nix-store --query --references $PWD/shell.drv) \ --indirect --add-root $PWD/shell.dep Best, Andreas [1]: https://gist.github.com/aherrmann/51b56283f9ed5853747908fbab907316 [2]: https://nixos.org/wiki/Development_Environments#Making_a_Persistent_nix-shell_Environment ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] NIX_ENFORCE_NO_NATIVE
On Thursday 09 June 2016 10:51:30 Vladimír Čunát wrote: > On 06/09/2016 10:17 AM, Andreas Herrmann wrote: > >> IIRC its' conditioned on $NIX_ENFORCE_NO_NATIVE exactly in order > >> not to be filtered out when run outside a nix builder. IMO that's a > >> good default. Hydra isn't the only reason; many devs use remote > >> builds or nix-copy-closure. > > How would you nix-copy-closure something that was compiled outside of > > nix? Maybe, we're talking about different things. What I mean is > > using a GCC that was built by Nix directly on the command-line > > You seem to have missed my main point, so let me be more clear. > > 1) Outside nix builder $NIX_ENFORCE_NO_NATIVE is *not* active unless you > explicitly set it, i.e. -march=native *is* passed to the underlying GCC. > I re-tested that now by reading code and using (2). Thanks for clarifying that. I did indeed not get that point. I do all my development within nix-shells nearly all the time. I should have mentioned that in the beginning, sorry. It seems that it would be difficult to distinguish gcc being called from within a nix-builder to it being called on the command-line within a nix-shell. > 2) If you set NIX_DEBUG=1 those omission won't be silent: > $ env NIX_ENFORCE_NO_NATIVE=1 NIX_DEBUG=1 g++ [...] > skipping impure path -march=native > [...] > > (Yes, in this case it's not very friendly to call it a path.) I did not know about that flag. Thanks for mentioning it. I still think that a more verbose kind of warning would be preferable in that particular case. I guess for now I will have to develop a habit of running my nix builds with NIX_DEBUG=1 after an update of the package set to see if any compiler flags have changed. Best, Andreas ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] NIX_ENFORCE_NO_NATIVE
Thanks for your response. On Wednesday 08 June 2016 22:54:04 Vladimír Čunát wrote: > On 06/08/2016 04:14 PM, Andreas Herrmann wrote: > > Especially, when being run outside of a nix-builder. > > IIRC its' conditioned on $NIX_ENFORCE_NO_NATIVE exactly in order not to > be filtered out when run outside a nix builder. IMO that's a good > default. Hydra isn't the only reason; many devs use remote builds or > nix-copy-closure. How would you nix-copy-closure something that was compiled outside of nix? Maybe, we're talking about different things. What I mean is using a GCC that was built by Nix directly on the command-line $ gcc -O3 -march=native -o thing thing.c Or, through some build-system: $ make > If you do want -march or -mtune in mkDerivation, you may either set > NIX_ENFORCE_NO_NATIVE = false; [...] Sure, and I'm glad that it's an easy fix. What I'm taking issue with is that it was a quiet change, and that I'm not notified in any way that part of my compiler flags are being ignored. I only found out by chance when looking at the disassembly of some executable while profiling and optimizing. I noticed that there were no AVX instructions to be found, even though they where generated before with -march=native. My suggestion would be to add a warning. That would seem to work well for both cases: If you're cross-compiling or remote building then you probably don't want to set -march=native and a notification would help to fix these cases. If you're compiling on the target machine and want automatic architecture specific optimization then -march=native is there specifically for that purpose. It being silently ignored is, IMHO, surprising and unexpected behavior and not really helpful. > [...] or even better, specify it explicitly, > e.g. -march=btver1 Well, I think, whether that's better or not depends on the use case. If you have different targets but can compile directly on the target then why not use the automatic detection? That way it's one thing less to worry about. Again, I agree that removing -march=native is a good thing on Hydra. It makes sense to me as the default behavior when building things with nix. It's certainly better than unexpected breakage after a nix-copy-closure. But, I think in any case the user should be notified. Best, Andreas ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] NIX_ENFORCE_NO_NATIVE
Hi everyone, I'm using Nix to manage some of my numerics applications. I am compiling the programs specifically on and for a certain platform and am using GCC's `-march=native` flag so that the compiler can apply all the platform specific optimizations it knows. (SSE, AVX, etc.) I noticed that this doesn't really work as expected, anymore. Many of the platform features are not detected automatically. While digging through nixpkgs I found that there is an environment variable `NIX_ENFORCE_NO_NATIVE` that defaults to `1` which means that the `-march=native` flag is going to be ignored. The reasoning is to prevent impurity. This flag was introduced this march [1]. Now, I understand that this is important for packages that are compiled on Hydra for the binary cache. But, I think it's not a good thing for GCC to quietly ignore the `-march=native` flag. Especially, when being run outside of a nix-builder. Maybe a warning would be appropriate in that case, or even in general? After all, people usually have a good reason to set `-march=native`. In those cases performance tends to be important. Best, Andreas [1]: https://github.com/NixOS/nixpkgs/commit/0c6db0ca48612f18e5c2b744dfa385ba8eecc950 ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] GCC 5, Matplotlib, and tcmalloc don't work together
Thanks for your response, Vladimir. On Wednesday 01 June 2016 16:01:58 Vladimír Čunát wrote: > On 06/01/2016 11:30 AM, Andreas Herrmann wrote: > > Maybe it's just a bad idea to mix allocators and I was lucky before. > > Deallocating memory that was allocated by a different allocator > certainly means trouble, and immediate crash is the better option IMO. My understanding was the following, and please correct me if I'm wrong. I thought that Python was using the allocator it was compiled with (glibc) and my C++ module would be using the allocator it was compiled with (tcmalloc). pybind11 (similar to boost-python) would tell Python to call back into the C++ module to release any object coming out of that module such that the C++ runtime could take care to call the destructor and properly free the memory through, in that case, tcmalloc. That way those two allocators would coexist without getting into each other's way. So far, I never had any allocator related issues and no memory leaks so I assumed that the above picture was correct. Now, the issue only appears with matplotlib.pylab which, through the qt backend, introduces its own C++ module. Previously, they were compiled with two different compilers with different C++ ABI versions (pre C++11, and post C++11). Now, after the update they're both compiled with the same GCC and C++ ABI version. I assume that this causes some conflict. It's probably more a C++ runtime issue than a Python issue - I don't know. For me it's not a big deal. The Python module only does some post-processing where performance is not critical. Thanks to Nix I can have two separate versions of my C++ library without any effort. > BTW, tcmalloc compares itself to glibc-2.3; IIRC since then the default > allocator has improved significantly, in particular in multi-threaded usage. I found notable speedups between tcmalloc and glibc malloc (whichever version was in nix at that point) about a year ago for my use-case. Since then I haven't checked again. Best, Andreas ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] GCC 5, Matplotlib, and tcmalloc don't work together
Hi, I'm writing this in case anyone else encounters a similar issue. I have a C++ library that is primarily used by C++ executables, but is also used by a Python package. I noticed that ever since the update of nixpkgs to GCC 5 (determined by git bisect) Python would crash if I first imported matplotlib.pylab (with any of the Qt backends) and then my own C++ module. If I imported them in the opposite order then no error would occur. (The error message was: `src/tcmalloc.cc:278] Attempt to free invalid pointer 0x154b870`) The C++ module was using tcmalloc from gperftools [1]. Today I found out, that the Python issue goes away if I do not use tcmalloc but instead just standard glibc malloc for the C++ module. Previously I was explicitly using GCC 5 for the C++ module, but was using the Python 2.7 stack that came with nixpkgs, presumably compiled with GCC 4.9. In that configuration tcmalloc was not causing any issues. I'm not opening an issue on github about this because I don't have any short example code that I can share to reproduce the issue. Also, I don't know if there is anything to fix about this. Maybe it's just a bad idea to mix allocators and I was lucky before. Let me know if you think it deserves an issue anyway. The C++ library is made available to Python using pybind11 v1.7 [2]. All this was happening on 64 bit Linux. Best, Andreas [1]: http://goog-perftools.sourceforge.net/doc/tcmalloc.html [2]: https://github.com/pybind/pybind11 ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Haskell Infrastructure - Nix-Shell with Hoogle from cabal2nix
Hi William, On Tuesday 16 February 2016 09:57:10 William Casarin wrote: > wrt. the hoogle question, there's a bunch of solutions written up here: > > http://stackoverflow.com/questions/27728838/using-hoogle-in-a-haskell-development-environment-on-nix Thanks for pointing me there, the answer by Ben was exactly what I was looking for. I think, for now at least, I prefer the per-project version over a global override. > The way I do this is create a custom environment for my haskell tools like so: > > https://github.com/jb55/nix-files/blob/8a47989a1091910fdba202f7def235ddcca6b726/nixpkgs/config.nix#L62-L77 > > and > > https://github.com/jb55/nix-files/blob/8a47989a1091910fdba202f7def235ddcca6b726/nixpkgs/config.nix#L43-L46 > > Then I can just nix-env -i haskellTools to put it in my path, and > nix-env -e haskellTools to > remove it from my path when I'm done. You could even automate this > with a script. Thanks for sharing your config. That's an interesting approach. I'll try it out once I've found out which packages I commonly need. Best, Andreas ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Haskell Infrastructure - Nix-Shell with Hoogle from cabal2nix
Hi Peter,
Thanks for your response.
On Monday 15 February 2016 19:17:52 Peter Simons wrote:
> > In order to create a development environment, where I have
> > `cabal-install` available as well, I can write the following Nix
> > file: [...].
>
> why don't you just put inherently compiler-agnostic tools like
> cabal-install, stack, alex, happy, etc. into your user's profile or
> system profile? Then all your nix-shells will have access to them.
I'm not using Nixos, but Nix in single user mode on a host system. I use Nix
for a bunch of different things. To avoid any collisions I've developed a habit
of keeping as many things as possible in separate nix-shells rather than in my
user profile. In the case of cabal it probably wouldn't hurt to put it in my
user profile. Here it was meant as an example of how to add things to the
development environment.
On Sunday 14 February 2016 14:48:54 Andreas Herrmann wrote:
> shell.nix
> ~~~
> { nixpkgs ? import { }, compiler ? "default" }:
> let
>
> inherit (nixpkgs) pkgs;
> haskellPackages = if compiler == "default"
>
> then pkgs.haskellPackages
> else pkgs.haskell.packages.${compiler};
>
> inherit (pkgs.haskell.lib) addBuildTools;
> pkg = import ./pkg.nix { inherit nixpkgs compiler; };
>
> in
> (addBuildTools pkg (with haskellPackages; [ cabal-install ])).env
> ~~~
>
>
> # My Question
>
> How could I instead get a development environment with a Hoogle package
> database of all the dependencies. I know that I could manually create the
> following Nix file and use it with `nix-shell ./shell-manual.nix`.
>
> shell-manual.nix
> ~~~
> { nixpkgs ? import { }, compiler ? "default" }:
> let
>
> inherit (nixpkgs) pkgs;
> haskellPackages = if compiler == "default"
>
> then pkgs.haskellPackages
> else pkgs.haskell.packages.${compiler};
>
> ghc = haskellPackages.ghcWithHoogle (pkgs: with pkgs; [
>
> base bytestring cabal-install
>
> ]);
>
> in
> pkgs.stdenv.mkDerivation {
>
> name = "dummy";
> buildInputs = [ ghc ];
> shellHook = "eval $(egrep ^export ${ghc}/bin/ghc)";
>
> }
> ~~~
>
> However, then I would have to manually maintain an extra dependency list. Is
> there any way of transforming the Nix-Shell environment produced by
> `pkg.env`
> in `shell.nix` above into one that adds a Hoogle database of all the
> included
> packages?
In any case, the question about a Hoogle database remains. I find it very
useful in situations where I don't have a reliable Internet connection at hand.
(train/plane/etc.)
Best, Andreas
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] Haskell Infrastructure - Nix-Shell with Hoogle from cabal2nix
Hi,
I'm trying to understand how to use the Haskell-infrastructure properly.
Suppose I have an existing Haskell project with the following cabal file:
test.cabal
~~~
name:test
version: 0.1.0.0
license: BSD3
license-file:LICENSE
build-type: Simple
cabal-version: >=1.10
executable test
main-is: Main.hs
build-depends: base, bytestring
default-language:Haskell2010
~~~
I can use `cabal2nix` to generate a Nix derivation for it. The tool produces
the following file:
default.nix
~~~
{ mkDerivation, base, bytestring, stdenv }:
mkDerivation {
pname = "test";
version = "0.1.0.0";
src = ./.;
isLibrary = false;
isExecutable = true;
executableHaskellDepends = [ base bytestring ];
license = stdenv.lib.licenses.bsd3;
}
~~~
And in order to build that package I can create the following Nix file:
pkg.nix
~~~
{ nixpkgs ? import { }, compiler ? "default" }:
let
inherit (nixpkgs) pkgs;
haskellPackages = if compiler == "default"
then pkgs.haskellPackages
else pkgs.haskell.packages.${compiler};
in
haskellPackages.callPackage ./. { }
~~~
And then build the package with `nix-build ./pkg.nix`.
In order to create a development environment, where I have `cabal-install`
available as well, I can write the following Nix file:
shell.nix
~~~
{ nixpkgs ? import { }, compiler ? "default" }:
let
inherit (nixpkgs) pkgs;
haskellPackages = if compiler == "default"
then pkgs.haskellPackages
else pkgs.haskell.packages.${compiler};
inherit (pkgs.haskell.lib) addBuildTools;
pkg = import ./pkg.nix { inherit nixpkgs compiler; };
in
(addBuildTools pkg (with haskellPackages; [ cabal-install ])).env
~~~
# My Question
How could I instead get a development environment with a Hoogle package
database of all the dependencies. I know that I could manually create the
following Nix file and use it with `nix-shell ./shell-manual.nix`.
shell-manual.nix
~~~
{ nixpkgs ? import { }, compiler ? "default" }:
let
inherit (nixpkgs) pkgs;
haskellPackages = if compiler == "default"
then pkgs.haskellPackages
else pkgs.haskell.packages.${compiler};
ghc = haskellPackages.ghcWithHoogle (pkgs: with pkgs; [
base bytestring cabal-install
]);
in
pkgs.stdenv.mkDerivation {
name = "dummy";
buildInputs = [ ghc ];
shellHook = "eval $(egrep ^export ${ghc}/bin/ghc)";
}
~~~
However, then I would have to manually maintain an extra dependency list. Is
there any way of transforming the Nix-Shell environment produced by `pkg.env`
in `shell.nix` above into one that adds a Hoogle database of all the included
packages?
Best, Andreas
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Haskellng - enable profiling
Thanks for pointing me there.
On Tuesday 10 March 2015 10:45:00 Peter Simons wrote:
Andreas Herrmann writes:
| if profiling
| then mypackage.overrideScope (self: super: {
| inherit stdenv; # Why this line is required is beyond me. =/
| mkDerivation = drv: super.mkDerivation (drv // {
enableLibraryProfiling = true; });
| })
| else mypackage
Inheriting 'stdenv' is required because of
https://github.com/NixOS/nixpkgs/issues/6192.
Best regards,
Peter
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Haskellng - enable profiling
I just had the same problem. I have no idea whether it's the right way to do
it, but I got it working in the following way:
Assuming that your package description is in `default.nix` (as created by
cabal2nix) add the following two files to the same directory:
mypackage.nix:
| { profiling ? false }:
|
| with (import nixpkgs {}).pkgs;
|
| let
|mypackage = haskellngPackages.callPackage ./. {};
| in
|
| if profiling
| then mypackage.overrideScope (self: super: {
| inherit stdenv; # Why this line is required is beyond me. =/
| mkDerivation = drv: super.mkDerivation (drv // {
enableLibraryProfiling = true; });
| })
| else mypackage
shell.nix:
| { profiling ? false }:
|
| with (import nixpkgs {}).pkgs;
| with (import nixpkgs/pkgs/development/haskell-modules/lib.nix { inherit
pkgs; });
|
| (overrideCabal (import ./mypackage.nix { inherit profiling; }) (drv: {
|buildTools = with haskellngPackages; [ cabal-install ghc-mod yesod-bin ];
| })).env
Now you can build your package with
nix-build --arg profiling true mypackage.nix
And you can open a nix-shell with
nix-shell --arg profiling true
I hope this helps.
Best, Andreas
On Monday 02 March 2015 00:25:26 Carlo Nucera wrote:
Hi all, I'd like some help to build a haskell package I'm developing
while transitively enabling profiling. I'm trying to do the thing
described in
http://lists.science.uu.nl/pipermail/nix-dev/2015-January/015620.html
What I'd like here is a demonstration on how to do that directly in
the definition on myPackage (see below).
Here is, more or less, my .nixpkgs/config.nix:
| { pkgs }: {
|allowBroken = true;
|allowUnfree = true;
|
|haskellPackageOverrides = self: super: {
| ghci-ng = self.callPackage /path/to/ghci-ng/default.nix {};
| myPackage = self.callPackage /path/to/myPackage/default.nix {};
|};
|
|packageOverrides = pkgs: with pkgs; rec {
|
| haskellEnv = haskellngPackages.ghcWithPackages cabalPackages;
|
| cabalPackages = hp: with hp; [
|cabal2nix
|cabal-install-bundle
|# Other libraries
|
|myPackage
| ];
|};
| }
The goal here is to build myPackage with profiling enabled
(transitively for his dependencies). If you have other advises on this
file, feel free to share.
Thanks
Carlo
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Tracking down dependencies
On Wednesday 28 January 2015 13:23:08 Thomas Bereknyei wrote: During a recent `nixos-rebuild switch --upgrade` I noticed gtk+ being downloaded from cache. Is there a quick way to discover what is causing that to be needed? [Rather than going through all of my configuration.nix packages one-by-one?] This question is also generic, is there a way to do this in general? You can use `nix-store -q --referrers /nix/store/your-derivation.drv` to list every derivation that depends on your-derivation. Best, Andreas ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] gcc vs gfortran
Hi, I was just looking into gfortran and noticed that I don't quite understand that package. gfortran48 is defined as an override of gcc48 with the following arguments: langFortran = true; langCC = false; langC = false; To me this suggests that the package gfortran comes with a Fortran compiler, but does not offer a C, or a C++ compiler. But, if I look at the built derivation I find that there are programs cc, cpp, gcc, and g++ alongside f77, and gfortran. Is that intentional, and if so why? The reason why I'm asking is this: I would like to use nix to install programs for people to use outside of nix. E.g. nix-build -A nixpkgs.gcc48 'nixpkgs' -o /global/path/gcc48 People can then add /global/path/gcc48/bin to their path and use gcc. However, since there is overlap between the gcc and gfortran packages this could be problematic if people want to use both. Best, Andreas ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] gcc vs gfortran
On Wednesday 10 December 2014 12:12:01 Eelco Dolstra wrote: Looks like a bug. Okay, I opened an issue about it: https://github.com/NixOS/nixpkgs/issues/5287 Best, Andreas ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Is anyone using Julia with PyPlot?
On Monday 01 December 2014 18:19:38 Bjørn Forsman wrote: Might be that matplotlib runs with wrong (non-gui) default backend...? At least I had some issues with that when packaging gnuradio. It does indeed `plt.get_backend()` returns agg. Thanks for that hint. I just noticed, that I misread the README of PyPlot. If I do the following it works properly: julia using PyCall # PyCall, not PyPlot julia pygui(:qt) # :wx works too julia using PyPlot julia plot([1,2,3],[1,2,4]) # Displays plot window On Tuesday 02 December 2014 09:53:38 Michael Raskin wrote: Haven't yet launched X since your last email, but have you tried pygui(:wx)? With the above it's working fine in the cli. The only remaining issue is with the IJulia notebooks. Plots are only shown inline if I explicitly display the figure object. I would expect this to happen the same way as in IPython just after a call to plot, as advertized in [1]. On Monday 01 December 2014 21:50:07 Domen Kožar wrote: Most probably related to https://github.com/NixOS/nixpkgs/issues/4500 I haven't tried GTK3, yet. But, for me plotting works fine in Python with qt4, and wx: import matplotlib matplotlib.use(Qt4Agg) # Or WXAgg import matplotlib.pylab as plt plt.plot([1,2,3],[1,2,4]) plt.show() # Shows the plot window I'll take a closer look at it later. [1]: http://nbviewer.ipython.org/url/jdj.mit.edu/~stevenj/IJulia%20Preview.ipynb ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] Is anyone using Julia with PyPlot?
Hi everyone,
I'm having an issue with the Julia Nix-expression[1]. I am trying to use it
together with the Julia package PyPlot[2] to plot figures with matplotlib. If I
try it in the command-line interface, then no plots show up. I only get a
textual representation of the figure objects. However, when I use
`savefig(tmp.png)` then the plot is correctly saved to the output file. In
the IJulia[3] notebook no inline plots are displayed after calls to `plot`, or
`title`, only the textual object representations. However, if I call `gcf()` to
retrieve the current figure object, then the plot is correctly displayed inline
in the notebook. Within python everything works fine: Plots in the commandline
`ipython --pylab=qt`, as well as in the notebook with `%matplotlib inline` are
displayed as expected. I would like to get Julia to display plots in gui
windows from the command-line, and inline in IJulia notebooks.
Please note, I am not sure if this is an issue with the Julia Nix-expression,
or with the Julia package `PyPlot`. However, before posting an issue on the
PyPlot project I would like to try and make sure that it's not an issue with my
use of Nix. Hence, my question: Is anyone here using Julia as a Nix-package to
create plots with matplotlib through PyPlot in the cli, or in an IJulia
notebook? If so, how did you get it working?
I am running Julia within a nix-shell of the following expression:
{ nixpkgs ? nixpkgs
, system ? builtins.currentSystem }:
with (import nixpkgs { inherit system; });
let py = pkgs.pythonPackages; in
stdenv.mkDerivation {
name = dummy;
src = ./empty;
buildInputs = with py; [
stdenv python readline ipython julia pyqt4 matplotlib
];
}
Best, Andreas
[1]:
https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/compilers/julia/0.3.3.nix
[2]: https://github.com/stevengj/PyPlot.jl
[3]: https://github.com/JuliaLang/IJulia.jl
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Is anyone using Julia with PyPlot?
Personally, I use Julia wrong (general purpose data crunching, little
enough plotting to just use gnuplot).
Nothing wrong with that. ;)
I currently lack time for learning it, but I guess I could do some quick
strace-ing if given a step-by step instruction (doing X expected Y
observed Z).
Thank you, I appreciate that. I don't know what to expect down to the syscall
level. But, I can give steps and expectations on the user-interaction level.
However, these are my expectations after reading the PyPlot readme file, and
according to my memory of how it used to work about half a year ago outside of
Nix. Unfortunately, the opensuse package for Julia seems to be broken, so I
cannot easily test another Julia installation outside of Nix. (I'm running Nix
on openSUSE 13.1 64bit)
So, here it goes: Take the file `default.nix` listed below, and start a
nix-shell with it. In that nix-shell execute
sh$ julia
julia Pkg.add(PyPlot)
julia using PyPlot
julia pygui(:qt); using PyPlot
julia plt.isinteractive() # Should print true
julia plt.plot([1,2,3], [1,2,4])
julia plt.show() # Should show a gui window with a line-plot.
``` default.nix
{ nixpkgs ? nixpkgs
, system ? builtins.currentSystem }:
with (import nixpkgs { inherit system; });
let py = pkgs.pythonPackages; in
stdenv.mkDerivation {
name = dummy;
src = ./empty;
buildInputs = with py; [
stdenv python readline ipython julia pyqt4 matplotlib
];
}
```
Best, Andreas
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] Error in travis that I can't reproduce on my system
Hi, I'm working on a PR[1] to add the blitz++ library[2] to Nixpkgs. However, I'm baffled by an error that travis produces [2, line 822]. The package builds fine on my Nix in openSUSE as well as on NixOs in a virtualbox. It seems to be some issue with autotools where the configure script puts a literal `no` into a `#define` that should just be left empty if the feature is not available. Any ideas how I could fix this? Is there something different about the tool-chain used on travis? Best, Andreas [1]: https://github.com/NixOS/nixpkgs/pull/5062 [2]: https://sourceforge.net/projects/blitz/ [3]: https://travis-ci.org/NixOS/nixpkgs/builds/41625038#L822 ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] My new email signature
On Tuesday 18 November 2014 00:45:07 Luca Bruno wrote: Does it look good on your email clients? :) KMail 4.11.5 here: http://i.imgur.com/65WDu9p.png ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Vim conceal, cscope and other features
Hi,
Is there an easy way to
enable them (I know I can copy the nix file and edit it to my hearts
content, but I was hoping there is a better way)?
You could use `overrideDerivation` and hotfix the configureFlags. Grep for it
in nixpkgs/pkgs to see a few example.
There is an instructive one in pkgs/servers/x11/xorg/overrides.nix
let inherit (lib) overrideDerivation; in
# ...
darwinOtherX = overrideDerivation xorgserver (oldAttrs: {
# ...
configureFlags = oldAttrs.configureFlags ++ [
--disable-xquartz
--enable-xorg
--enable-xvfb
--enable-xnest
--enable-kdrive
];
# ...
});
Best, Andreas
On Wednesday 12 November 2014 00:17:07 Richard Wallace wrote:
Hey all,
I just realized that vim in nix isn't configured with the conceal and
cscope features enabled. Any reason for that? Is there an easy way to
enable them (I know I can copy the nix file and edit it to my hearts
content, but I was hoping there is a better way)?
Thanks,
Rich
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] nix-multi-user installation script
Hi, Thanks for having a look at this! On Thursday 06 November 2014 13:17:57 Wout Mertens wrote: On Tue Oct 28 2014 at 10:15:03 PM Andreas Herrmann andreas...@gmx.ch wrote: I would like to ask you guys to have a look at this script and tell me what you think of it. Do you think it could be valuable to add something like this as an automated installer to the official Nix distribution similar to the one for single-user mode? Definitely, although I think it would be best to have it as script to run after one already installed Nix. That way, existing installs can just be converted. That is an interesting idea. The current single user install script installs nix for a non-root user. How would you go about transforming such an installation to a multi-user nix? Just chown it to root, or are there more subtle issues to think about? (In particular the default-profile...) I did come up with a few specific questions during the course of this: * The nix-daemon is not daemonizing itself. Why is that? On debian this is not a problem thanks to `start-stop-daemon`. On CentOS I ended up writing a wrapper script. Self-daemonizing daemons are not so wonderful. It's nicer to have a watchdog daemon doing the background running and logging, and it makes the program a little simpler. Besides, you can daemonize anything from the shell simply by closing stdin and running in the background: $ run_as_daemon $LOGFILE 21 - echo $! $PIDFILE The shell does the right thing, disassociating the child when it exits etc. Nice, I had forgotten about `-`. I should modify my wrapper accordingly. The reason I chose a wrapper instead of doing the above right in the init-script is, that CentOS offers a few standard shell functions which do things like print that nice green `[ OK ]` and such. * The chroot build feature seems to require a statically linked bash. Compiling this (plus some dependencies) can take quite some time. Do you think a static bash would be a viable addition to `nixpkgs`? Not really - you just need to make sure all the libraries are in the chroot. I would think that it just works on Linux, where everything is linked only to /nix/store (unless the build got tainted). On Darwin the stdenv isn't pure yet but even then the required libraries can be copied. On Thursday 06 November 2014 14:41:25 Peter Simons wrote: it just works on NixOS, because the system auto-generates an appropriate /etc/nix/nix.conf file that includes all bits required by bash into the chroot environment. [...] On other Linux systems, this needs to be configured manually. So, coming back to this. Do you think the static bash would be a good package to add? On Thursday 06 November 2014 13:17:57 Wout Mertens wrote: Note that the chroot isn't really required to build things and in its current form it slows down builds. See https://github.com/NixOS/nix/issues/179 . As I understand this the build seems to be slowed down by all the bind mounts. But, shouldn't a non-trivial package spent far more time on actual building than on bind mounting? I just think that the benefit in purity due to chroot outweights the cost of the bind mount. * Is there a way to add globally visible package overrides but still use the nix-channel? Absolutely, that's what packageOverrides in the nixpkgs configuration is for. If a binary build is available, it will be used, otherwise you'll do the build locally. Thanks, then let me follow up on this: How can I configure global packageOverrides that will be visible to all users; yet, still allow them to add their own packageOverrides on top of that? I.e. where does the global equivalent of `~/.nixpkgs/config.nix` go on a non-NixOS multi-user Nix? Best, Andreas ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] deploying nix-built software to non-nix linux systems
There was a post here recently about using proot to install Nix in $HOME [1]. I haven't used it myself, but I could imagine that you could install a closure of your package and provide a wrapper script which configures the environment and uses proot to run your app. [1]: https://nixos.org/wiki/How_to_install_nix_in_home_%28on_another_distribution%29 Best, Andreas On Tuesday 04 November 2014 10:24:15 Luca Bruno wrote: On 04/11/2014 00:36, Anders Papitto wrote: Hello, I'm interested in using nix/nixos to build an application with its full set of dependencies, and then deploy it to non-nixos machines as a self-contained package (up to and including the system libraries), without root privileges. Ideally I would like to not require that any nix tools be available on the target system. The perfect workflow would be something like # on any linux system $ wget https://some/path/to/my-app.tar.gz $ tar -xf my-app.tar.gz $ ./my-app/bin/my-app # run the application - it should not try to access anything outside ./my-app Is this possible? I haven't been able to find any resources on deploying to a non-nix systems. You don't have to install nix to run packages on other systems, you can simply copy the closure of it. However usually packages are stored into /nix, which requires root access. You may try something like fakechroot or proot. ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] ghc-mod depends on Wayland
Hello everyone,
Recently, I tried to install the ghc-mod package. I was surprised to see that
nix-env intended to install a _large_ amount of dependencies down to quite
unexpected ones such as Wayland.
I dug into the dependency tree:
nix-store -qR --tree $(nix-instantiate -A haskellPackages.ghcMod
'nixpkgs')
/nix/store/0b7bmyslyidn7iq7bcmxzgxqa0wcd786-haskell-ghc-mod-ghc7.8.3-5.1.1.0-shared.drv
+---/nix/store/gzgrizf2xw537hnnbycbnpbxzrwddxjw-emacs-24.4.drv
| +---/nix/store/lp9z54f5snvrdnvlih815m28lxb5080l-librsvg-2.36.4.drv
| | +---/nix/store/l89khlm6piiilivpnx618w13cfqqj223-cairo-1.12.16.drv
| | |
+---/nix/store/ib90dbdwvijmv44v7ij7ghs1md5nfrf2-mesa-noglu-10.2.6.drv
| | | |
+---/nix/store/gl09hmxv11fvsz2dkr6s6i5xsbvp2c55-wayland-1.6.0.drv
As it turns out this comes from Emacs and the problem is fixed with a simple
package override:
{ pkgs }: {
packageOverrides = self: with pkgs; {
emacs = self.emacs.override { withX = false; withGTK = false; };
}
}
However, I'm wondering if this really is the intended default behavior for that
package. Wouldn't it be better to have ghc-mod depend on a nox-emacs by default?
Best, Andreas
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] ghc-mod depends on Wayland
Thanks for pointing this out. I agree, I think a build-time dependency on emacs-nox, and no run-time dependency on emacs would be a great solution. Best, Andreas On Monday 03 November 2014 11:26:12 Peter Simons wrote: Please see https://github.com/NixOS/nixpkgs/issues/4561. Peter ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] fetching tarballs recursively
On Thursday 30 October 2014 12:45:09 Sergey Mironov wrote: Replacing --import/--export instead of --dump/--restore also leads to problems. This time, the error is $ cat dump | nix-store --import importing path `/nix/store/n69kz335c800n2b6izsjbw60rlnvp87c-binutils-2.23.1.tar.bz2' error: imported archive of `/nix/store/n69kz335c800n2b6izsjbw60rlnvp87c-binutils-2.23.1.tar.bz2' lacks a signature what kind of signature does nix-store expect? I'd be interested in a good answer to this as well. I encountered the same problem in my multi-user install script [1] for a closure of a static bash. Anyway, my (probably hackish) work-around was to bypass the daemon and write directly to the nix-store by executing NIX_DAEMON= nix-store --import ... But, I assume that this won't work on NixOS because the store is mounted read-only. Best, Andreas [1]: https://github.com/aherrmann/nix-multi-user-install ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] nix-multi-user installation script
Hello everyone, in the past two weeks or so I have been experimenting with Nix multi-user mode. My goal was to be able to reliably install Nix in multi-user mode on a CentOS 6.5 machine. The result of this was a bash script which pretty much automates the whole process [1]. The script is still a bit rough around the edges, but I got it working repeatedly and reliably on fresh CentOS 6.5 installations. I would like to ask you guys to have a look at this script and tell me what you think of it. Do you think it could be valuable to add something like this as an automated installer to the official Nix distribution similar to the one for single-user mode? Unfortunately, CentOS 6.5 comes with fairly old packages. Therefore, it turns out to be next to impossible to compile Nix from sources. (You'd have to compile the rest of the world as well). The Fedora rpm packages can't be used either - for the same reason. So, I ended up using the distribution agnostic binary package of Nix that is also used for the single-user-mode install script. To that I added features necessary for multi-user-mode (daemon, and config), and chroot builds, and some `/etc/profile.d` settings to make Nix easy to use for all users. I did come up with a few specific questions during the course of this: * The nix-daemon is not daemonizing itself. Why is that? On debian this is not a problem thanks to `start-stop-daemon`. On CentOS I ended up writing a wrapper script. * The chroot build feature seems to require a statically linked bash. Compiling this (plus some dependencies) can take quite some time. Do you think a static bash would be a viable addition to `nixpkgs`? * Is there a way to add globally visible package overrides but still use the nix-channel? * Would it be possible to make a non-root user the admin of a multi-user nix installation? I.e. a user who can can do `nix-channel --update`. Or does this explicitely require root? Let me give some credit. I did greatly profit from previous work done by Sander van der Burg [2], on the Nix wiki [3], on the single-user mode install script, and of course on the Nix documentation. Thanks a lot to the corresponding authors. [1]: https://github.com/aherrmann/nix-multi-user-install [2]: http://sandervanderburg.blogspot.ch/2013/06/setting-up-multi-user-nix-installation.html [3]: https://nixos.org/wiki/Installing_Nix_on_Debian ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] nix on compute cluster?
Thanks for your answer. On Tuesday 14 October 2014 09:16:54 Alexander Kjeldaas wrote: On Mon, Oct 13, 2014 at 10:14 PM, Andreas Herrmann andreas...@gmx.ch wrote: On Saturday 11 October 2014 23:36:36 you wrote: For a compute grid, where no graphics, no desktop, no kde etc. are compiled, a full NixOS build is in the hours, not days. I would simply compile this on a single server and rdist, No need to over-engineer this. Well, the reason I'm worried is that if any, or at least a number of users, could cause recompilation of a fairly large number of libraries, then this could make the login node quite sluggish. And I remember the compilation of libatlas alone taking a whole night on my machine. I guess it depends on how many changes to the nixpkgs tree will happen. I have never used any pre-cooked binaries and I'm generally using low-end Xeons. I'm only using a basic set of development tools though (gcc, ghc, bunch of haskell libraries, emacs ...). I do track nixpkgs HEAD, so I recompile quite a lot, but it it just something I do in a terminal. One pattern that I noticed for myself, is that I often need to add new features to existing libraries. E.g. SGE mode for MPI, or MPI mode for hdf5. If many packages depend on these, then this will cause a large number of rebuilds. But maybe we should just try a simple multi-user install and see how it works. One could hope that most of the heavy compilation would be done in a few days, after which it's only small compile jobs that are already done regularly by the users. Just compare the scenarios between a normal RHEL installation and Nix. A developer asks for libatlas version Y and version X is installed. 2 weeks of debate ensures. Using Nix, you just start compiling. Ok, so i takes the night, about the same time it would take to discuss it on the mailing list. I'm afraid, things are far less organized than you might think. Also, please note that I'm not the admin (we have one) I'm just an enthusiastic user... Unfortunately, the current state of affairs is that - in many cases - just everyone compiles their own version of a library they need in their home-directory. So, everyone has their own version of boost, and a number of people even have their own gcc. I think that's a bit of a waste of time and space. Fortunately, atlas scale libraries exist globally. I'm not sure why your users would trigger a recompilation. If I were you I would have one blessed nipkgs tree, and if they wanted a particular version of some library, do it as a commit to that tree. If you have every user have their own patched nixpkgs tree, that might cause problems, yes. Currently, I have nix installed in single user mode. And I am using it to handle all sorts of weird little dependencies that can't be found in any distributions repos. These are quite domain specific to the work that we are doing, and different people depend on different packages. However, for me nix has been an incredible productivity booster because it became so easy to swap packages and their dependencies around. By now I do also write nix-packages for my own projects, it makes it far easier to share code among them. Finally, the myEnvFun feature is very useful to encapsulate dependencies, and run jobs in different environments simultaneously. However, these benefits are only available if you can have your own `~/.nixpkgs/config.nix` with your own package overloads. But, once a user has that he can cause any arbitrary recompilation. One question: Why do you thing rdist would be better than a (cached) NFS mounted nix-store? NFS with a cache should work fine. It's been many years since I used NFS and at the time only IRIX had a decent NFS cache. That's a fair point. I have no idea how well the NFS cache works on CentOS. I would just hope that it shouldn't be so bad in a scenario where the only time-critical operations are reads from the nodes and writes are seldom. Best, Andreas ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] nix on compute cluster?
Hi, How would you go about bringing the benefits of Nix to the users of a compute cluster? Assume the following cluster: A login node, a file-system node, and a number of compute nodes. All nodes run on a recent CentOS and are fairly homogeneous. The fs node holds all user data and some common libraries. Its storage is nfs mounted on all other nodes. Users ssh into the login node, write and compile some code, then they use the Sun Grid engine (sge) to submit compute jobs, and once these are finished they copy the results on their workstations and are happy. There are subgroups of users with fairly exotic software requirements. These are not available in any package repositories, and the cluster admin doesn't have the time to install and maintain them. So, currently, most of these users just compile everything themselves in their home-directory, which is a huge waste of time, and storage space. I would like to suggest Nix to the admin as a way to let these user-subgroups manage their own packages, but that in a well organized manner, that avoids redundant work, and storage. But, I'm not sure how exactly that should work. There are a few constraints: 1. Unfortunately, NixOS/nixops is not an option. This will have to work with the currently installed cluster OS. 2. Compilation should not put too much load on the login node. Ideally, build jobs would be referred to the compute nodes. 3. Build jobs on the compute nodes should be managed by the sge. 4. (Some) users should be allowed to initiate builds, and use their own overloads of packages, and extra packages. 5. Some impurity is necessary. Be it for things that are hard to package (e.g. intel compiler), or for global state (mpi jobs). My question to you: Do you think this is possible to achieve (within a reasonable time-frame), and how would you do it? Here's what I have in mind so far (please feel free to take it apart if you think there is a better way): Have a nix-store on the file-server, nfs mount that on all nodes (cached). The login node runs the nix-daemon. Builds are deferred to the grid-engine (how?) which are executed on the compute nodes, and store the results on the nfs mounted nix-store. Users would use `nix-env` on the login node to install software into their profile. This profile should be visible on all nodes, so that jobs can use those libraries and tools in the nix-profile. Things like myEnvFun should allow running jobs in different software environments simultaneously. Best, Andreas ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] nix on compute cluster?
Thank you for the detailed answer. On Friday 10 October 2014 15:32:52 Wout Mertens wrote: I think you could do this. You would set it up so the nix server does the compiles and the grid runs distcc. See the wiki, the raspberry pi page has explanations about distcc. Oh, I didn't know that this worked outside of NixOS. I just can't find any details on how to integrate distcc with sge. Do you have any experience with that? Note that only one node can write to nix store at the same time due to the db. That's interesting. The multi-user nix manual says that you can have multiple build users on one machine. Why does that work on one machine, but not spread over several machines? I would assume that you need to connect to some kind of db daemon anyway? Another option is to have private nix stores on all nodes but nfs mount all of them under the remote stores directory. That way nix-store will fetch missing packages from the remotes and store them locally. At least, that's my understanding. I don't think I understand this. By private, do you mean one per user? And how do we mount all of them under one directory, do you mean some layered fs? As for the Intel compiler, that could be a challenge, but right now we have several gcc versions and clang, so it's not impossible. You can decide on a per-package basis which compiler to use. Yes, my main concern are things like license keys, or - even worse - license servers. I don't know how to approach these things in nix. As long as you can mix your nix environment with an external environment this is not an issue. So, as a simple example, you could use nix to manage your version of boost, but use an intel compiler that is installed outside of nix for user code. Not sure how mpi would influence state, can you elaborate? I meant this on a per job basis. In the case of mpi jobs the sge will decide which nodes your job will be distributed over. This information will be passed by a mixture of config files and environment variables. Hence, such jobs would not work inside a pure nix-shell, because the environment needs to be passed through. I don't think it's a big issue. Just something to keep in mind. Best, Andreas ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] nix on compute cluster?
On Friday 10 October 2014 17:49:20 Wout Mertens wrote: On Fri, Oct 10, 2014 at 4:34 PM, Andreas Herrmann andreas...@gmx.ch wrote: On Friday 10 October 2014 15:32:52 Wout Mertens wrote: I think you could do this. You would set it up so the nix server does the compiles and the grid runs distcc. See the wiki, the raspberry pi page has explanations about distcc. Oh, I didn't know that this worked outside of NixOS. I just can't find any details on how to integrate distcc with sge. Do you have any experience with that? No sorry, but I'd just install distcc as a daemon on all nodes or else use sge for the distribution somehow... pretend that they're super long running batch jobs... I think an on demand approach would fit better for the current usage pattern than a permanently running build server. Looking at distcc more closely, it seems like a too low level approach. E.g. it wouldn't cover clang, or icc (once in nix) compilations. The sun grid engine allows for scheduled interactive remote shells. It also allows to submit any random shell script, or sequence of commands as a batch job. Shouldn't there be a way to make nix run its builders as such jobs? To my understanding userHook is too late. It's only executed inside the builder shell, and can't be used to open a remote shell session that executes the builder, correct? Best, Andreas ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] library with python bindings
On Wednesday 08 October 2014 18:37:00 Damien Cassou wrote: On Wed, Oct 8, 2014 at 5:04 PM, Andreas Herrmann andreas...@gmx.ch wrote: How do I tell Nix to fix `PYTHONPATH` accordingly when this package is loaded, and also to wrap python accordingly? I think you have to use things like wrapProgram $out/bin/yourprogram --prefix PYTHONPATH : ... and propagatedBuildInputs Look at the existing derivations. That was what I needed. I had looked at wrapPythonProgram beforem, which didn't work, because these programs are actually bash scripts, that call the python interpreter. Now, I added `makeWrapper` to `buildInputs`, and added postFixup = '' for i in $out/bin/*; do wrapProgram $i --prefix PYTHONPATH : $PYTHONPATH done ''; to my derivation and everything works. Thanks a lot. ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] issue with channels and hydra
On Wednesday 08 October 2014 11:43:57 Eelco Dolstra wrote: It hasn't finished building, and it hasn't been mirrored to cache.nixos.org yet. That's why you're not getting binaries. Ah, sorry, I didn't know that. The current channel version at http://nixos.org/channels/nixpkgs-unstable/ is b36095578cd4b12c976c95c18610ddb7d38dcf28, so if you use that, you should get binaries. Hmm, that one still has the old bash 4.2 p49, which doesn't work for me, because my host system has newer patches and the corresponding environment crashes the nix bash [1]. I think I need at least p51. I'll try to get along with 14.04-small for now. Is there a way to make nix download from a not fully evaluated jobset? I read somewhere on the wiki that you can, but it didn't say how. [1]: http://pastebin.com/bARMTwq5 ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] issue with channels and hydra
On Wednesday 08 October 2014 16:05:43 Eelco Dolstra wrote: On 08/10/14 13:17, Andreas Herrmann wrote: Is there a way to make nix download from a not fully evaluated jobset? You can use hydra.nixos.org as a binary cache server by passing the option --option extra-binary-caches http://hydra.nixos.org But this should only be done as a last resort, because hydra.nixos.org is slow and overloaded already :-) Thanks, Eelco, I'll keep that in mind. Btw, I checked once more, and the unstable channel is now on 0b23b5b which has bash-4.2-p50 which actually works on my system. So, for now I'm good. =) Best, Andreas ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] library with python bindings
Hi, suppose I have a package that is mainly a C++ library and also written in C++ and built with CMake and GCC. Yet, it also provides a python library that is compiled and written to `$out/lib/python2.7/site-packages/...` as a mix of shared object files and python scripts. How do I tell Nix to fix `PYTHONPATH` accordingly when this package is loaded, and also to wrap python accordingly? I looked around in nixpkgs but wasn't able to identify a certain pattern to follow in this case. Best, Andreas ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] bash: error importing function definition for `BASH_FUNC_module()'
Dear Nixers,
for a few days now I'm having issues with a number of programs that are managed
by nix. E.g. gcc 4.9, or nix-prefetch-git.
When I try to run these commands I get the following error message:
$ nix-prefetch-git
/nix/store/7k7wanhlkwzwaj55dirf9r0291z02hng-bash-4.2-p49/bin/bash: error
importing function definition for `BASH_FUNC_module()'
I suspect that this is related to the recent bash vulnerability [1].
I am using nixpkgs on openSUSE 13.1. openSUSE has all recent updates installed.
And nixpkgs is on the current checkout of master.
My system bash-version is `4.2.47(1)-release (x86_64-suse-linux-gnu)`. The
nixpkgs bash-version is `4.2.48(1)-release (x86_64-unknown-linux-gnu)`.
The environment variable (bash function) `BASH_FUNC_module` is introduced by
the `Modules` package [2] and reads:
$ env
# ...
BASH_FUNC_module()=() { eval
`/usr/share/Modules/$MODULE_VERSION/bin/modulecmd bash $*`
}
# ...
This thing is deeply nested in the global system configuration. Hence, it is
not sufficient to just wipe the corresponding environment variable, because the
next instance of bash is just going to read it from the global configuration
again.
I don't understand why this issue appears. I thought that both nixpkgs, and
openSUSE had the same necessary security patches for bash. Why are the openSUSE
and the nixpkgs bash versions in conflict?
Best,
Andreas
[1]: https://access.redhat.com/articles/1200223
[2]: http://modules.sourceforge.net/
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Moving some of the NixOS setup to nixpkgs
I have actually been thinking about a possible use case of this for a while. I just never found the time to try and see if it's possible. The idea would be to describe your vim configuration (plugins, their settings, and global config) with nix and then be able to transfer it to other systems through e.g. nix-copy-closure. Each plugin could be its own package, and have additional config options. And parts of your vim config could be individual packages that you can pick and swap around. Switching vim configurations with profiles, or env-func could also be a quite useful feature. E.g. if you want to have a certain vim plugin for one code-base, and another one for another code-base, and those two plugins are incompatible... Best, Andreas On Friday 03 October 2014 13:02:37 Vladimír Čunát wrote: On 10/03/2014 12:54 PM, Luca Bruno wrote: I think it's pointless wrapVim for nixos and non-nixos. For non-nixos you can have a ~/.vimrc, and for system-wide /etc/vimrc. Yeah, the argument was something like that, I think... The difference is that config inside profile is more flexible. No really strong reasons, but IMO useful enough to avoid removal from nixpkgs. (E.g. not global, included in nix-copy-closure transfers, easy to switch among multiple ones per-terminal.) Vladimir ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Moving some of the NixOS setup to nixpkgs
Hi, I'm using nixpkgs outside NixOS, and I have to agree that there are a few things that could be a little bit better. I'll just list a few things that I had to work around. * Curl seems to work fine, except for when I'm using nix-prefetch-url. I haven't figured out what the actual issue is. However, a workaround is to set `NIX_CURL_FLAGS=-k`. * MANPATH is not set by the default nix.sh profile script. So, I need to set it myself. It would be great if there was some automation for such environment variables. E.g. also INFOPATH... * Related to the above. I needed to manually install locals, and configure the `LOCAL_ARCHIVE` variable. See [1] for details. Cheers, Andreas [1]: http://lists.science.uu.nl/pipermail/nix-dev/2009-May/002405.html ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] cross-compile to non-standard nix-store location
Hi,
I'm using nix in single user mode in a non-standard location
(${HOME}/nix/store) on a CentOS host on which I don't have root-access, and
which I don't want to flood with lots of build jobs. I'm looking for a good way
to cross-compile binaries on my laptop (nix in single user mode in standard
location /nix/store) which I can then export and install on the CentOS machine.
---
Here is my current approach (Thanks to another exchange on the mailing list a
while ago). It works, but feels a bit hacky, is hard to automate, and doesn't
offer hardware specific compiler flags:
I created a new user on my laptop, which has the same name, and home as my user
on the CentOS machine. That user has its own nix installation, which was
configured to use the non-standard store path. I open a login shell in that
user's name, and use nix-build to build whatever package I want to install on
the CentOS machine. Finally, I use nix-store export/import to install it.
---
Is there a way to skip the extra user on my laptop? I tried a cross-compilation
configuration. However, it doesn't seem to pick up the non-standard nix-store
location, and actually fails to compile gcc4.9.
Here is the cross-compile config:
let
pkgsFun = import nixpkgs;
pkgsNoParams = pkgsFun { };
in
pkgsFun {
crossSystem = {
config = i686-pc-linux;
arch = x86_64;
libc = glibc;
withTLS = true;
float = hard;
platform = pkgsNoParams.platforms.pc64 // {
kernelMajor = 2.6;
};
gcc.arch = corei7-avx;
};
config = pkgs: {
packageOverrides = pkgs: {
nix = pkgs.nix.override {
storeDir = /home/user/nix/store;
stateDir = /home/user/nix/store;
};
};
};
}
I had to apply the following patch to nixpkgs because the linker name was
missing for this platform:
diff --git a/pkgs/build-support/gcc-wrapper/default.nix
b/pkgs/build-support/gcc-wrapper/default.nix
index 8e8b0b9..9a014b7 100644
--- a/pkgs/build-support/gcc-wrapper/default.nix
+++ b/pkgs/build-support/gcc-wrapper/default.nix
@@ -70,6 +70,7 @@ stdenv.mkDerivation {
(if stdenv.cross.arch == arm then ld-linux.so.3 else
if stdenv.cross.arch == mips then ld.so.1 else
if stdenv.lib.hasSuffix pc-gnu stdenv.cross.config then ld.so.1
else
+ if stdenv.cross.arch == x86_64 then ld-linux-x86-64.so.2 else
abort don't know the name of the dynamic linker for this
platform);
};
When I try to build gcc4.9 with `nix-build -A gcc49.crossDrv beo.nix` then it
fails with this message:
checking for .preinit_array/.init_array/.fini_array support... no
configure: error: Need linker with .init_array/.fini_array support.
builder for
`/nix/store/hfix8jkynaj5r32kavyg2n1r5mp38092-glibc-2.19-i686-pc-linux.drv'
failed with exit code 1
cannot build derivation
`/nix/store/x3z0v3szgv4wd9icdbbwk2x6zhxky35m-gcc-wrapper-4.9.1-i686-pc-linux.drv':
1 dependencies couldn't be built
error: build of
`/nix/store/x3z0v3szgv4wd9icdbbwk2x6zhxky35m-gcc-wrapper-4.9.1-i686-pc-linux.drv'
failed
It seems that something is wrong with this linker configuration.
And it doesn't seem to pick up the non-standard nix-store path, as this is only
an override for the nix package.
---
What is the best way to achieve what I'm trying to do? Is there a way to avoid
the extra user on my laptop?
Thanks!
Andreas
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] automatically mount vboxsf
Hi Nicolas,
On Saturday 06 September 2014 20:25:43 Nicolas Pierron wrote:
Have you tried listing it as an entry of the fileSystem of the vm, such as:
fileSystems./host_home = {
fsType = vboxsf;
device = hostHome;
};
Thanks for the hint. I got it working with the following config:
fileSystems./host_home = {
fsType = vboxsf;
device = hostHome;
options = ro; # If you want it read-only
noCheck = true; # fsck would be pointless.
};
Additionally, I had to add the vbox guest additions into the scope:
system.fsPackages = [
config.boot.kernelPackages.virtualboxGuestAdditions
];
Best,
Andreas
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] automatically mount vboxsf
Dear Luca, On Monday 08 September 2014 10:57:31 Luca Bruno wrote: system.fsPackages = [ config.boot.kernelPackages.virtualboxGuestAdditions ]; I suggest you to use services.virtualbox.enable = true. Doesn't nixops set this by default when the deployment is virtualbox? At least, I haven't manually set this service to enabled before, but the guest additions were installed anyway. For instance, I was able to manually mount the vboxsf volume once the machine had booted. I had the feeling, that the package just wasn't there, yet when the system tried to automatically mount the vboxsf volume at boot-time. So, adding it to fsPackages should somehow make sure that the package is available at boot-time. Or am I wrong here? Best, Andreas ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] automatically mount vboxsf
Hi everyone,
I've been playing around with nixops and virtualbox deployments for a bit.
nixops offers a configuration option to forward a host folder into the guest os
like so:
deployment.virtualbox.sharedFolders = {
hostHome = { hostPath = /home; };
};
However, inside the virtual machine these are not available immediately. These
folders have to be mounted manually by executing the following command as root:
mount.vboxsf hostHome /host_home
I was wondering if there was some way to automate this process. I couldn't find
vboxsf anywhere in the supported file-systems. Is there another way to
automatically mount it when the virtual machine boots?
Best,
Andreas
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Can't SSH into a root account
Dear Mateusz, Isn't SSH default configured to only accept public-key login for root? First thing I found on google about it: http://askubuntu.com/questions/449364/what-does-without-password-mean-in-sshd-config-file The option `permitRootLogin` should fix it: https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/networking/ssh/sshd.nix#L117 Or have you tried that already? Best, Andreas On Friday 22 August 2014 11:22:13 Mateusz Kowalczyk wrote: Hi, I while ago I put NixOS on another computer. I set it all up and it's great except for one problem: I can't SSH into the root account on it. See below for some attempts: I want to SSH from @lenalee to @yuuki. It just keeps rejecting my password even know I know it's correct: I demonstrate that by SSHing into a regular user on that box and then using ‘su’. This also shows that it's not a problem with SSH access in general, just to that user. The only thing I'm not showing here is SSHing into root@yuuki from regular user @lenalee because I set up login with a public key for convenience. This also that I can actually log into the user, just not with the password. I hoped that this would go away after I stopped messing with the users at install time but it has now been weeks and multiple reboots c and the problem persists. Below is an annotated session dump. I especially point your attention to the last part. ## -- Try logging into the root user, fail. [root@lenalee:/home/shana]# ssh root@yuuki Password: Password: ^C [root@lenalee:/home/shana]# exit -- Log into the regular user there. [shana@lenalee:~]$ ssh shana@yuuki Password: Last login: Fri Aug 22 11:10:18 2014 from lenalee -- …and switch to the root that way… [shana@yuuki:~]$ su Password: -- …which works so the password is correct. [root@yuuki:/home/shana]# exit [shana@yuuki:~]$ logout Connection to yuuki closed. [shana@lenalee:~]$ su Password: -- Keep trying, keep failing. That second way of failure and prompting -- makes me suspicious. [root@lenalee:/home/shana]# ssh root@yuuki Password: Password: Password: root@yuuki's password: Permission denied, please try again. root@yuuki's password: Permission denied, please try again. root@yuuki's password: [root@lenalee:/home/shana]# exit ## It'd be great if someone could offer some advice. ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] daemon in httpd subservice
Hi guys, Unfortunately, I still haven't solved this problem. To be a bit more precise on the issue: I have a httpd subservice which contains a PHP based web-app (TTRSS) that uses postgresql as a database. All that works fine. However, it also requires a daemon to run in order to update the RSS/Atom feeds on a regular basis. (I would prefer the daemon over a cron job). You can observe my current progress here [1]. My idea is the following: I have one file (`ttrss-service.nix`) that contains the httpd subservice, and another file (`ttrss-update.nix`) that contains the update daemon. Unfortunately, the update daemon needs to know the exact installation location of ttrss (`ttrssRoot`), and that one depends on the configuration of the subservice. (Because the config file needs to be stored in that directory as well.) So, the issue boils down to one question: How can I extract the installation path (`ttrssRoot`) from the httpd subservice? That question appears at [2] in the code. Of course, if you have a better idea how to tackle TTRSS, please let me know. On a side note: I first experimented with a simpler case on how to get a daemon running in NixOS. You can find the results in [3]. It works. However, if you think that this is the wrong way to configure a service in NixOS please let me know. Best regards, Andreas [1]: https://github.com/aherrmann/nixops-ttrss-test/tree/a899bc571b8c581f85cf7d92270e168d67535374 [2]: https://github.com/aherrmann/nixops-ttrss-test/blob/a899bc571b8c581f85cf7d92270e168d67535374/ttrss.nix#L39 [3]: https://github.com/aherrmann/nixops-deamon-test/tree/a495c8930267c488217a7b3a4dd9722006638903 ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] nixops mediawiki example
Dear Mateusz, Now I understand that the example has to be a bit more complicated because it is a configuration on two different machines. And the database server cannot rely on the httpd subservice, since that one is on a different machine. However, it took me some time to understand this. I've created a pull-request, that adds another mediawiki example on just one single machine. It might be helpful to others... Best, Andreas On 19 July 2014 21:03, Mateusz Kowalczyk fuuze...@fuuzetsu.co.uk wrote: On 07/19/2014 10:34 AM, Andreas Herrmann wrote: Digging deeper into the mediawiki package showed that it already has a db-setup script. The fix was to simply remove it from my config. The configuration in [1] works now. Simply type `http://vm-ip/wiki` into the browser and start editing. To help future users it might be a good idea to update the mediawiki example in [2]. What do you think? Best, Andreas [1]: https://github.com/aherrmann/nixops-mediawiki-test/tree/e947c2d38211ab8a1b1c948b82deff723c2ef74c [2]: https://github.com/NixOS/nixops/blob/master/examples/mediawiki.nix ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev Documentation is always good and even better if it's up to date. Send a pull request and I'm sure someone will take it. -- Mateusz K. ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] daemon in httpd subservice
@Bit Shift: Thanks for your response. [...] but could you not define a crontab entry that invokes the updater [...] The TTRSS documentation lists a couple of different ways to update the feeds. A crontab entry is indeed one of the possible options. However, the documentation lists the daemon as the recommended way of updating the feeds. And, since we have full control of the NixOS machine, I don't see why we shouldn't be able to spawn that daemon. Best, Andreas ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] daemon in httpd subservice
Hi,
I am now trying to get Tiny-Tiny-RSS [1] to work in NixOS via nixops. TTRSS
is an RSS/Atom aggregator and reader that you can run on your own server
and then access in multiple ways to read your news. (Including an android
app).
I implemented it as a subservice for httpd much like the mediawiki module.
And, the app itself is working fine. However, TTRSS comes with a php-cgi
script to update the feeds, that is intended to be run as a daemon [2]. And
this poses the following question:
Is it possible to define a service (daemon) from within an httpd
subservice, and how can I do that? Or, will I have to define a separate
(top-level) service that is manually activated in the machine configuration
and somehow depends on the TTRSS installation path? Which way would you
recommend, or prefer?
My current attempt looks like this:
-- machine configuration contains
services.httpd = {
enable = true;
adminAddr = ad...@example.com;
extraSubservices = singleton
{ function = import ./ttrss-service.nix;
siteHostName = 192.168.56.101;
};
};
-- subservice configuration contains
systemd.services.ttrssUpdate = {
description = TTRSS feed-update daemon;
wantedBy = [ multi-user.target ];
after = [ postgresql.service httpd.service ];
serviceConfig = {
ExecStart = ${php}/bin/php ${ttrssRoot}/update.php --daemon;
User = serverInfo.serverConfig.user;
Restart = always;
};
};
Best, Andreas
[1]: http://tt-rss.org/redmine/projects/tt-rss/wiki
[2]: http://tt-rss.org/redmine/projects/tt-rss/wiki/UpdatingFeeds
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] SSH-keys in fetchgit
Hi, Turns out that my ssh config was correct. But, you need to specify the url for bitbucket in the following form: url = g...@bitbucket.org:user/my_project.git; It works now. Thanks again for your help. Best, Andreas On 16 July 2014 00:27, Andreas Herrmann andreas...@gmx.ch wrote: Dear Shea, Thanks for pointing me there. I gave it a shot but cannot get it to work. I set-up an ssh config directory for nix in `$HOME/.nixpkgs/ssh/`. In there is a key file pair without passphrase `id_rsa`, `id_rsa.pub`. The public key is configured as a deployment key on the bitbucket repo. I changed the derivation to use `fetchgitPrivate` instead of fetchgit. And it also showed the expected trace if I do not specify `ssh-config-file`. So, it should at least call the right fetcher. However, if I run the following command-line. NIX_PATH=ssh-config-file=$HOME/.nixpkgs/ssh/:$NIX_PATH nix-build --show-trace -A my_package Then I get connection time-outs to bitbucket these derivations will be built: /nix/store/8j2i6a33bfikqbvaz4wg7adaxnpyp9j5-git-export.drv /nix/store/jhc3l1iib93afj2xf9sc2r9y9amv4lb3-my_package-git-619cfcd.drv building path(s) `/nix/store/icd6rr1xhk79fl17w24f2c6czqb5nvhz-git-export' exporting git://bitbucket.org/user/my_package.git (rev 718dfcd9f6895e9e879aa112c348e4e0d2ba90d4) into /nix/store/icd6rr1xhk79fl17w24f2c6czqb5nvhz-git-export Initialized empty Git repository in /nix/store/icd6rr1xhk79fl17w24f2c6czqb5nvhz-git-export/.git/ fatal: unable to connect to bitbucket.org: bitbucket.org[0: 131.103.20.167]: errno=Connection timed out bitbucket.org[1: 131.103.20.168]: errno=Connection timed out I also tried explicitely configuring an identity file in a host-block inside `$HOME/.nixpkgs/ssh/config`. It didn't help. Do you know what I'm missing? Best regards, Andreas On 15 July 2014 17:39, Shea Levy s...@shealevy.com wrote: See the fetchgitPrivate function: https://github.com/NixOS/nixpkgs/blob/a4cf5e903b7782062d62489b23fa5021d1ac0862/pkgs/top-level/all-packages.nix#L308-L326 On Tue, Jul 15, 2014 at 12:09:22PM +0200, Andreas Herrmann wrote: Dear Nix-ers, I am using Nix to manage and deploy my own work projects, which do depend on each other. These projects, however, are not open source, so their sources are not publicly available on the internet. I was wondering if it was possible in some way to configure Nix (or fetchgit) to use ssh keys when fetching sources from non-public repositories. (E.g. private repositories on bitbucket with deployment keys) Please note, that I am using Nix/Nixpkgs in single-user mode on a host OS (openSUSE, and CentOS). I tried adding my deployment keys to the system-wide ssh config in `/etc/ssh/`, and the per user config `~/.ssh/`, both to no avail. Also, at least on the CentOS machine I do not have root access. So, whatever solution there is will have to work from within my home directory. My current (hacky) solution is to hard code the path to the repository in the `src` attribute of the derivation in the form `url = file:///home/user/src/project;`. And then ship it to the CentOS machine by using `nix-store --export`, and `nix-store --import`. However, this is hardly a satisfactory solution, and as you can imagine it doesn't scale very well with the number of packages. Best, Andreas ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] SSH-keys in fetchgit
Dear Nix-ers, I am using Nix to manage and deploy my own work projects, which do depend on each other. These projects, however, are not open source, so their sources are not publicly available on the internet. I was wondering if it was possible in some way to configure Nix (or fetchgit) to use ssh keys when fetching sources from non-public repositories. (E.g. private repositories on bitbucket with deployment keys) Please note, that I am using Nix/Nixpkgs in single-user mode on a host OS (openSUSE, and CentOS). I tried adding my deployment keys to the system-wide ssh config in `/etc/ssh/`, and the per user config `~/.ssh/`, both to no avail. Also, at least on the CentOS machine I do not have root access. So, whatever solution there is will have to work from within my home directory. My current (hacky) solution is to hard code the path to the repository in the `src` attribute of the derivation in the form `url = file:///home/user/src/project;`. And then ship it to the CentOS machine by using `nix-store --export`, and `nix-store --import`. However, this is hardly a satisfactory solution, and as you can imagine it doesn't scale very well with the number of packages. Best, Andreas ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] SSH-keys in fetchgit
Dear Shea, Thanks for pointing me there. I gave it a shot but cannot get it to work. I set-up an ssh config directory for nix in `$HOME/.nixpkgs/ssh/`. In there is a key file pair without passphrase `id_rsa`, `id_rsa.pub`. The public key is configured as a deployment key on the bitbucket repo. I changed the derivation to use `fetchgitPrivate` instead of fetchgit. And it also showed the expected trace if I do not specify `ssh-config-file`. So, it should at least call the right fetcher. However, if I run the following command-line. NIX_PATH=ssh-config-file=$HOME/.nixpkgs/ssh/:$NIX_PATH nix-build --show-trace -A my_package Then I get connection time-outs to bitbucket these derivations will be built: /nix/store/8j2i6a33bfikqbvaz4wg7adaxnpyp9j5-git-export.drv /nix/store/jhc3l1iib93afj2xf9sc2r9y9amv4lb3-my_package-git-619cfcd.drv building path(s) `/nix/store/icd6rr1xhk79fl17w24f2c6czqb5nvhz-git-export' exporting git://bitbucket.org/user/my_package.git (rev 718dfcd9f6895e9e879aa112c348e4e0d2ba90d4) into /nix/store/icd6rr1xhk79fl17w24f2c6czqb5nvhz-git-export Initialized empty Git repository in /nix/store/icd6rr1xhk79fl17w24f2c6czqb5nvhz-git-export/.git/ fatal: unable to connect to bitbucket.org: bitbucket.org[0: 131.103.20.167]: errno=Connection timed out bitbucket.org[1: 131.103.20.168]: errno=Connection timed out I also tried explicitely configuring an identity file in a host-block inside `$HOME/.nixpkgs/ssh/config`. It didn't help. Do you know what I'm missing? Best regards, Andreas On 15 July 2014 17:39, Shea Levy s...@shealevy.com wrote: See the fetchgitPrivate function: https://github.com/NixOS/nixpkgs/blob/a4cf5e903b7782062d62489b23fa5021d1ac0862/pkgs/top-level/all-packages.nix#L308-L326 On Tue, Jul 15, 2014 at 12:09:22PM +0200, Andreas Herrmann wrote: Dear Nix-ers, I am using Nix to manage and deploy my own work projects, which do depend on each other. These projects, however, are not open source, so their sources are not publicly available on the internet. I was wondering if it was possible in some way to configure Nix (or fetchgit) to use ssh keys when fetching sources from non-public repositories. (E.g. private repositories on bitbucket with deployment keys) Please note, that I am using Nix/Nixpkgs in single-user mode on a host OS (openSUSE, and CentOS). I tried adding my deployment keys to the system-wide ssh config in `/etc/ssh/`, and the per user config `~/.ssh/`, both to no avail. Also, at least on the CentOS machine I do not have root access. So, whatever solution there is will have to work from within my home directory. My current (hacky) solution is to hard code the path to the repository in the `src` attribute of the derivation in the form `url = file:///home/user/src/project;`. And then ship it to the CentOS machine by using `nix-store --export`, and `nix-store --import`. However, this is hardly a satisfactory solution, and as you can imagine it doesn't scale very well with the number of packages. Best, Andreas ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Problem with python distutils
Dear Mateusz,
Good point, I had just copied that from some other package.
It also works with `configurePhase = ;`.
Best, Andreas
On 3 July 2014 04:10, Mateusz Kowalczyk fuuze...@fuuzetsu.co.uk wrote:
On 07/02/2014 01:45 PM, Andreas Herrmann wrote:
Hi,
I found a work-around. The trick is to not use setuptools.
This can be achieved by setting `configurePhase=:;`. The installPhase
needs to be fixed as well.
installPhase = ''
mkdir -p $out/lib/${python.libPrefix}/site-packages
export
PYTHONPATH=$out/lib/${python.libPrefix}/site-packages:$PYTHONPATH
${python}/bin/${python.executable} setup.py install \
--install-lib=$out/lib/${python.libPrefix}/site-packages \
--prefix=$out
'';
Then it builds just fine.
Is that the best way to deal with this issue, or is there a better way?
Best, Andreas
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Why configurePhase=: and not (just empty string)?
--
Mateusz K.
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Problem with python distutils
Hi,
I found a work-around. The trick is to not use setuptools.
This can be achieved by setting `configurePhase=:;`. The installPhase
needs to be fixed as well.
installPhase = ''
mkdir -p $out/lib/${python.libPrefix}/site-packages
export
PYTHONPATH=$out/lib/${python.libPrefix}/site-packages:$PYTHONPATH
${python}/bin/${python.executable} setup.py install \
--install-lib=$out/lib/${python.libPrefix}/site-packages \
--prefix=$out
'';
Then it builds just fine.
Is that the best way to deal with this issue, or is there a better way?
Best, Andreas
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] Problem with python distutils
Hello everyone, I am trying to add a nix-expression for mpi4py. It is a wrapper module for the message passing interface for python. Unfortunately, the build fails with with a distutils error. (The error message is attached at the end of this mail.) When I googled the error I found a seemingly related issue in the setuptools issue tracker [1]. It seems to be a problem in how distutils and setuptools interact. A naive fix that I tried was to simply update setuptools. However, that failed quite badly. Setuptools seems to be a very complicated package and I wasn't able to upgrade it. (I got into circular dependencies with pytest.) So, I was wondering if anyone of you every came across this issue and knows how to fix it, or has an idea where to start looking for a solution. You can find my nix-expression for mpi4py under [2], and the corresponding call in [3]. Best, Andreas [1]: https://bitbucket.org/pypa/setuptools/issue/73/typeerror-dist-must-be-a-distribution [2]: https://github.com/aherrmann/nixpkgs/blob/hdf5mpi/pkgs/development/python-modules/mpi4py/default.nix [3]: https://github.com/aherrmann/nixpkgs/blob/hdf5mpi/pkgs/top-level/python-packages.nix#L113 -- error message: building 'mpi4py.dl' extension /nix/store/hpcv8grqy84b4ijq7ixgg7rq9bk0lh2x-openmpi-1.6.5/bin/mpicc -fPIC -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DHAVE_DLFCN_H=1 -DHAVE_DLOPEN=1 -I/nix/store/c2a5czy3lvs16gr6lhkjk2d0gj41zhg1-python-2.7.7/include/python2.7 -c src/dynload.c -o build/temp.linux-x86_64-2.7/src/dynload.o /nix/store/hpcv8grqy84b4ijq7ixgg7rq9bk0lh2x-openmpi-1.6.5/bin/mpicc -shared build/temp.linux-x86_64-2.7/src/dynload.o -L/nix/store/c2a5czy3lvs16gr6lhkjk2d0gj41zhg1-python-2.7.7/lib -Lbuild/temp.linux-x86_64-2.7 -Wl,-R/nix/store/c2a5czy3lvs16gr6lhkjk2d0gj41zhg1-python-2.7.7/lib -ldl -lpython2.7 -o build/lib.linux-x86_64-2.7/mpi4py/dl.so running build_exe Traceback (most recent call last): File setup.py, line 588, in module main() File setup.py, line 585, in main run_setup() File setup.py, line 459, in run_setup **metadata) File /tmp/nix-build-python2.7-mpi4py-1.3.1.drv-7/mpi4py-1.3.1/conf/mpidistutils.py, line 494, in setup return fcn_setup(**attrs) File /nix/store/fwabq51yh482v19qs7x1c619zp1yypmb-python2.7-distutils.cfg/lib/python2.7/site-packages/distutils/core.py, line 151, in setup dist.run_commands() File /nix/store/fwabq51yh482v19qs7x1c619zp1yypmb-python2.7-distutils.cfg/lib/python2.7/site-packages/distutils/dist.py, line 953, in run_commands self.run_command(cmd) File /nix/store/fwabq51yh482v19qs7x1c619zp1yypmb-python2.7-distutils.cfg/lib/python2.7/site-packages/distutils/dist.py, line 972, in run_command cmd_obj.run() File /nix/store/fwabq51yh482v19qs7x1c619zp1yypmb-python2.7-distutils.cfg/lib/python2.7/site-packages/distutils/command/build.py, line 127, in run self.run_command(cmd_name) File /nix/store/fwabq51yh482v19qs7x1c619zp1yypmb-python2.7-distutils.cfg/lib/python2.7/site-packages/distutils/cmd.py, line 326, in run_command self.distribution.run_command(command) File /nix/store/fwabq51yh482v19qs7x1c619zp1yypmb-python2.7-distutils.cfg/lib/python2.7/site-packages/distutils/dist.py, line 970, in run_command cmd_obj = self.get_command_obj(command) File /nix/store/fwabq51yh482v19qs7x1c619zp1yypmb-python2.7-distutils.cfg/lib/python2.7/site-packages/distutils/dist.py, line 846, in get_command_obj cmd_obj = self.command_obj[command] = klass(self) File /nix/store/fwabq51yh482v19qs7x1c619zp1yypmb-python2.7-distutils.cfg/lib/python2.7/site-packages/distutils/cmd.py, line 59, in __init__ raise TypeError, dist must be a Distribution instance TypeError: dist must be a Distribution instance note: keeping build directory `/tmp/nix-build-python2.7-mpi4py-1.3.1.drv-7' builder for `/nix/store/qwb90mn9cvdv5pzizvxag0a86agwpsip-python2.7-mpi4py-1.3.1.drv' failed with exit code 1 error: build of `/nix/store/qwb90mn9cvdv5pzizvxag0a86agwpsip-python2.7-mpi4py-1.3.1.drv' failed ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] szip unfree?
Dear Eelco, On 25 June 2014 13:59, Eelco Dolstra eelco.dols...@logicblox.com wrote: On 25/06/14 12:46, Louis Bettens wrote: This makes me wonder what /free/ means. We should clarify this in the docs, but I would say free according to https://www.gnu.org/philosophy/free-sw.html or http://opensource.org/docs/osd. I agree that a clarification in the docs would be a good idea. Going by the GNU definition SZip, in particular, offers all but freedom 0. The freedom to run the program as you wish, for any purpose. The OSD is a bit more complicated. But, SZip seems to violate the first item (Free Redistribution) as well. Concerning the license meta attribute: As I understand the definition in [1] we could actually apply `unfree-redistributatble` to SZip. Because, the license does not distinguish between source, or binary distribution. In any case you're not allowed to use it for commercial purposes. But, if that does not apply to you, then you are allowed to redistribute it modified, or unmodified, in source, or binary form. I.e. once you made sure that you're not using it for commercial purposes, you can at least profit from the hydra build service. Correct? Best, Andreas [1]: http://nixos.org/nixpkgs/manual/#sec-meta-license ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] szip unfree? (was: Re: pytables, numexpr, and pandas)
Dear Florian, dear Mateusz,
On 23 June 2014 22:58, Florian Friesdorf f...@chaoflow.net wrote:
szip unfree or not is an issue independent of your pull request. I
merged - thanks a lot!
Thanks, I'm glad I could add something.
I think you were right in not including the support by default.
Further, szip might have to set meta.license = unfree, so you need to
allow it eg. in ~/.nixpkgs/config.nix.
{
allowUnfree = true;
}
Opinions?
Yes, I think that would be appropriate.
One note, the hdf5 package in all-packages.nix had szip enable by
default. You need to pass szip = null. See my latest commit to master.
Okay, thanks, I didn't know that.
On 23 June 2014 23:56, Mateusz Kowalczyk fuuze...@fuuzetsu.co.uk wrote:
Some packages use an if expression for license field. For example:
license = if hdf5 == null then licenses.unfree else licenses.whatever
I don't know whether that's encouraged but it is a way to do it.
No, I don't think this is the right approach in this case. As far as I
know, a dependency does not propagate it's license to the software that
depends on it. That being said, if you want to use szip together with hdf5,
you will have to fulfull both licenses' requirements.
The hdf5 library itself has a license that I would call free. It permits
redistribution in source, or binary form, with or without modification,
basically only requiring that the original copyright notice and license is
shipped along with it [1].
Tables is licensed under a BSD license [2]. I think that one is generally
accepted as a free license.
Only szip has an unfree license, because it doesn't allow for commercial
use [3]. But whether you build hdf5 with or without szip doesn't change the
license of hdf5, or tables.
Best,
Andreas
[1]: http://www.hdfgroup.org/HDF5/doc/Copyright.html
[2]: http://www.pytables.org/moin/FAQ#WhatarePyTables.27licensingterms.3F
[3]: http://www.hdfgroup.org/doc_resource/SZIP/Commercial_szip.html
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] pytables, numexpr, and pandas
Dear Florian, Thanks for your response. On 23 June 2014 17:13, Florian Friesdorf f...@chaoflow.net wrote: Thx for reporting, I'm working on nixos python test suites and made a note to investigate. Good to know. Note, that I don't expect numexpr, or tables to work with pypy, because they both require cython, and last I checked it didn't play well with pypy. I might be wrong, though. The solution looks good to me. However, from a brief look it feels that one would want the default hdf5 package to be built with szip and zlib. What do you think? Well, my reason for not including szip by default was it's license. Quoting the hdf5-group page [1]: Licensing terms The version of Szip distributed with HDF products is free for non-commercial use, which may occur in two sets of circumstances: Non-commercial users may use the Szip software integrated with HDF products to both encode (compress) and decode (uncompress) data. This applies to educational and research applications. Commercial users may use the software to decode any data. Further, they may use the software in internal activities that do not involve or result in the development of an Szip-based software product. Commercial licenses are available for commercial users who wish to distribute an Szip-based software product or engage in commercial uses that are not allowed above. For further licensing information or to view a copy of the Szip copyright statement, see Commercial use terms and the copyright and license notice pertaining to Szip in HDF products. So, it's a license that requires some thought to figure out if you're allowed to use it or not. I thought it would be better not to include it by default, so that no one breaks the license unknowingly. I don't know what the Nix policy generally is, when it comes to licenses. Basically, I'm returning the question: Is it okay to include this package by default, or not? Best, Andreas [1]: http://www.hdfgroup.org/doc_resource/SZIP/ ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] pytables, numexpr, and pandas
I've finally gotten around to make that pull-request.
You can find it here:
https://github.com/NixOS/nixpkgs/pull/3056
Thanks for your help!
On 15 June 2014 22:11, Mateusz Kowalczyk fuuze...@fuuzetsu.co.uk wrote:
On 06/15/2014 09:09 PM, Andreas Herrmann wrote:
Thanks for your detailed answer.
On 15 June 2014 02:28, Mateusz Kowalczyk fuuze...@fuuzetsu.co.uk
wrote:
On 06/14/2014 11:17 PM, Andreas Herrmann wrote:
A) Is there some kind of test suite that I need to run on the
expressions
before I can send a pull request? (I can build and use them...)
Simply make sure it builds and runs on top of the current master. For
Python, check multiple Python versions. nix-build can help you build the
packages.
Okay, it runs on all python versions except pypy. Which is to be
expected.
However, the pypy branch seems to be somewhat broken. The build failed at
setuptools long before it reached my package.
C2) The test-suite of pytables takes quite some time and has a number
of
failures of the not-implemented type. Is it worth including that
test-suite or should I just leave that one out?
I would include any steps necessary to run the tests but after ensuring
that the failures aren't NixOS/nix specific, I'd set doCheck = false:
this makes it easier for you or someone else to turn on the tests on any
future, fixed versions. Make sure to let the maintainer know that there
are failing tests.
I dug a little deeper and found that the problem was actually rooted in
the
hdf5 package. pytables needs it to be built with zlib. I put that in, and
also fixed the szip package. However, both could be called optional
dependencies on hdf5. Do you know, how I can make pytables depend on hdf5
with a certain set of parameters? Or are these things handled through
comments and documentation?
To be more explicit: say hdf5 has the parameters `{ stdenv, fetchurl,
zlib
? null, szip ? null }`. How can I make pytables depend on hdf5 in such a
way, that it requires `zlib != null`, but doesn't care about `szip`?
I can't say for sure but I'd imagine you'd simply override it as usual.
Say you have pytables at
development/python-modules/pytables.default.nix: your entry to
python-modules.nix could look like
pytables = callPackage ../development/python-modules/pytables {
hdf5 = pkgs.hdf5.override { zlib = pkgs.zlib; };
};
python-modules.nix seems to have a whole section for packages defined
elsewhere so perhaps look at that bit. The style there seems to be using
import rather than callPackage however. I imagine it would look similar
to the below:
pytables = import ../development/python-modules/pytables {
inherit (pkgs) stdenv fetchurl whateverotherdeps;
inherit python numbexpr whateverotherpythondeps;
hdf5 = pkgs.hdf5.override { zlib = pkgs.zlib; };
};
Although this is a mere guess: I have not tested this.
If you have decided to put the package definition in all-packages.nix
then the first way should work fine, just drop pkgs. prefixes.
D1) Pandas works without pytables. However, without it, it cannot
import/export hdf5 archives. Should I add PyTables as a dependency to
Pandas, or should every user individually add both to his environment
if
he
needs both? Or is there a way to make it an optional dependency?
Optional dependencies are possible but I am unsure how they are done for
Python packages. While I don't know how to specify such optional
dependencies in python-packages.nix, you can simply create a
nix-expression outside of it. See pkgs/games/anki/default.nix for an
example of a Python package with optional dependencies. Provide a
sensible default because only the default way will be built by Hydra.
If you need Hydra to build the package in more ways then you should make
two separate entries in all-packages.nix with the different flag setups
but if there's only one common case then one should suffice.
I went to the pandas documentation and found out that they actually list
about a dozen optional dependencies. I came to the conclusion, that it is
probably easiest to not list them at all, since in the python world such
a
dependency doesn't require re-building, but just availability in the
module
search path. No reason to multiply the number of binaries if they end up
all being the same.
You're correct that there's probably no reason to list them all but if
the package ends up with different features depending on what's
available at build time then perhaps you should allow the user of the
package to toggle these dependencies on, even if they're off by default.
--
Mateusz K.
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] pytables, numexpr, and pandas
Thanks for your detailed answer.
On 15 June 2014 02:28, Mateusz Kowalczyk fuuze...@fuuzetsu.co.uk wrote:
On 06/14/2014 11:17 PM, Andreas Herrmann wrote:
A) Is there some kind of test suite that I need to run on the expressions
before I can send a pull request? (I can build and use them...)
Simply make sure it builds and runs on top of the current master. For
Python, check multiple Python versions. nix-build can help you build the
packages.
Okay, it runs on all python versions except pypy. Which is to be expected.
However, the pypy branch seems to be somewhat broken. The build failed at
setuptools long before it reached my package.
C2) The test-suite of pytables takes quite some time and has a number of
failures of the not-implemented type. Is it worth including that
test-suite or should I just leave that one out?
I would include any steps necessary to run the tests but after ensuring
that the failures aren't NixOS/nix specific, I'd set doCheck = false:
this makes it easier for you or someone else to turn on the tests on any
future, fixed versions. Make sure to let the maintainer know that there
are failing tests.
I dug a little deeper and found that the problem was actually rooted in the
hdf5 package. pytables needs it to be built with zlib. I put that in, and
also fixed the szip package. However, both could be called optional
dependencies on hdf5. Do you know, how I can make pytables depend on hdf5
with a certain set of parameters? Or are these things handled through
comments and documentation?
To be more explicit: say hdf5 has the parameters `{ stdenv, fetchurl, zlib
? null, szip ? null }`. How can I make pytables depend on hdf5 in such a
way, that it requires `zlib != null`, but doesn't care about `szip`?
D1) Pandas works without pytables. However, without it, it cannot
import/export hdf5 archives. Should I add PyTables as a dependency to
Pandas, or should every user individually add both to his environment if
he
needs both? Or is there a way to make it an optional dependency?
Optional dependencies are possible but I am unsure how they are done for
Python packages. While I don't know how to specify such optional
dependencies in python-packages.nix, you can simply create a
nix-expression outside of it. See pkgs/games/anki/default.nix for an
example of a Python package with optional dependencies. Provide a
sensible default because only the default way will be built by Hydra.
If you need Hydra to build the package in more ways then you should make
two separate entries in all-packages.nix with the different flag setups
but if there's only one common case then one should suffice.
I went to the pandas documentation and found out that they actually list
about a dozen optional dependencies. I came to the conclusion, that it is
probably easiest to not list them at all, since in the python world such a
dependency doesn't require re-building, but just availability in the module
search path. No reason to multiply the number of binaries if they end up
all being the same.
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] pytables, numexpr, and pandas
Hi,
I've created two new nix-expressions, that I need to use the Pandas package.
* PyTables is a python library for HDF5 storage. It's a different approach
than h5py in that it is not a pythonic wrapper of the C library, but rather
implements relational db alike features. Pandas depends on it to read/write
HDF5 files.
http://www.pytables.org/moin
* numexpr speeds up numpy ndarray expressions. PyTables depends on it.
https://github.com/pydata/numexpr
However, I have a few questions before I go ahead and send a pull-request.
A) Is there some kind of test suite that I need to run on the expressions
before I can send a pull request? (I can build and use them...)
B) Which branch should I send the pull-request to? Is master fine?
C1) Both packages have a test-suite. However, it's not integrated into `
setup.py`. I managed to get it running within the nix build process anyway.
The corresponding `checkPhase` is attached below. Is that an acceptable way
of doing this, or should I just set `doCheck = false;`?
C2) The test-suite of pytables takes quite some time and has a number of
failures of the not-implemented type. Is it worth including that
test-suite or should I just leave that one out?
D1) Pandas works without pytables. However, without it, it cannot
import/export hdf5 archives. Should I add PyTables as a dependency to
Pandas, or should every user individually add both to his environment if he
needs both? Or is there a way to make it an optional dependency?
D2) The latest release of Pandas is 0.14.0, the nix-pkgs version is 0.12.0.
Should I use this opportunity to bump that version number, or should this
go through a different channel?
I've read the contributing guides on the wiki (at least those that I could
find). I learned a lot from them, but the above questions remained.
Best,
Andreas
The numexpr `checkPhase` within a call to `buildPythonPackage`:
--
# Run the test suite.
# It requires the build path to be in the python module search path.
checkPhase = ''
${python}/bin/${python.executable} EOF
import sysconfig
import sys
import os
f = lib.{platform}-{version[0]}.{version[1]}
lib = f.format(platform=sysconfig.get_platform(),
version=sys.version_info)
build = os.path.join(os.getcwd(), 'build', lib)
sys.path.insert(0, build)
import numexpr
numexpr.test()
EOF
'';
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Binary packages for custom store path installs
I've actually had the same question, recently. I was planning to try a few things myself before I ask here. But, now that the discussion is there I'll just chime in... Given a case where you can't get the admin to give you `/nix`, and where you have space/compute constraints on the target, would it be possible to create the full nix install on your own local machine in the correct paths, and then just copy it to the target? Maybe a little like your own, small, on-demand hydra... Is there some info on this in the docs? The closest I could find was cross-compiling [1], but it doesn't seem to address the `/somepath/nix/store` issue. Best, Andreas [1]: https://nixos.org/wiki/CrossCompiling On 13 June 2014 06:22, Mateusz Kowalczyk fuuze...@fuuzetsu.co.uk wrote: In the Nix manual it says: ## It is best not to change the Nix store from its default, since doing so makes it impossible to use pre-built binaries from the standard Nixpkgs channels — that is, all packages will need to be built from source. ## I'd like to know why it is impossible. I don't understand why it matters in the slightest whether the store sits at /nix/store or somewhereelse/nix/store. Why does it even know the difference? Are there plans to improve on this? After all, everything would still be in the same relative location to the store. I'm not even sure if it actually means that it is in fact impossible with how nix is designed as opposed to ‘it's not currently implemented’. In environments where one only has regular user rights resources are often constrained, be it hard drive space, memory or computational power. Incurring the penalty of having to compile everything on top of that merely because the store is not sitting at the top of the file system is rather sub-optimal so it would be great if there was a solution. -- Mateusz K. ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] myEnvFun for IPython with Nix on openSuse host
Ok indeed I can reproduce it. Can you please report a bug in nixpkgs? https://github.com/NixOS/nixpkgs/issues/new Thanks for confirming it. Here's the link to the issue: https://github.com/NixOS/nixpkgs/issues/2818 Cheers! On 3 June 2014 14:53, Luca Bruno lethalma...@gmail.com wrote: On 01/06/2014 17:17, Andreas Herrmann wrote: Dear Luca, Thanks for your response. What commit of nixpkgs are you tracking? How do I find that out? I installed Nix using the install script, not the from source installation. So, I never manually checked out any package list. I ran the following commands. They didn't really do anything. In my interpretation that means, that my Nix installation should already be up-to-date. Correct? Ok indeed I can reproduce it. Can you please report a bug in nixpkgs? https://github.com/NixOS/nixpkgs/issues/new ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] myEnvFun for IPython with Nix on openSuse host
Hi everyone,
I am very new to Nix, so let me apologize in advance if this ends up being
a stupid question.
# First, some back story:
The reason I got interested in Nix is that I am doing scientific computing
on a number of different machines with self-made simulation software.
Hence, I need to keep my development environments in sync on these various
machines. A tiresome task that, so far, I am performing manually. I
understand that Nix is promising to help in such situations.
So, I went ahead and installed Nix in single user mode on my laptop which
is running on openSuse 13.1. That is, I used the install script [1], and
installed Nix under /nix. On the wiki I found a section [2] about managing
development environments, which introduces a thing called myEnvFun. It
worked out for a simple environment to use numpy:
numpy = pkgs.myEnvFun {
name = numpy;
buildInputs = with pkgs; let p27=python27Packages; in [
stdenv python
p27.readline
p27.numpy
];
};
I've installed it using `nix-env -i env-numpy`, and loaded it using `
load-env-numpy`, and everything seems to work fine.
$ load-env-numpy
env-numpy loaded
$ which python
/nix/store/qikwsrbsyx59ygcmb36p1rgmfclgg882-python-2.7.6/bin/python
$ python
Python 2.7.6 (default, Apr 21 2014, 11:09:09)
[GCC 4.8.2] on linux2
Type help, copyright, credits or license for more information.
import numpy.version
numpy.version.version
'1.7.1'
However, I would prefer to do develop using the IPython shell. So, I wrote
another environment that encorporates ipython:
ipython = pkgs.myEnvFun {
name = ipython;
buildInputs = with pkgs; let p27=python27Packages; in [
stdenv python
p27.readline
p27.ipython
p27.numpy
];
};
Installing it works just fine. Loading the environment on the other hand
doesn't.
# The Problem
I get a huge amount errors about gdk-bug when I load the ipython
environment defined above:
$ load-env-ipython
mkdir: cannot create directory ‘/lib/gdk-pixbuf-loaders-2.0/’:
Permission denied
# repeated 61 times
mkdir: cannot create directory ‘/lib/gdk-pixbuf-loaders-2.0/’:
Permission denied
/nix/store/1j8nx7r6a7s76hj14b7lgqij7yddmvw2-gdk-pixbuf-2.30.6/nix-support/setup-hook:
line 11: /lib/gdk-pixbuf-loaders-2.0//loaders.cache: No such file or
directory
mkdir: cannot create directory ‘/lib/gdk-pixbuf-loaders-2.0/’:
Permission denied
# repeated 17 times
mkdir: cannot create directory ‘/lib/gdk-pixbuf-loaders-2.0/’:
Permission denied
env-ipython loaded
Depsite the error messages, the environment seems to work.
$ ipython
Python 2.7.6 (default, Apr 21 2014, 11:09:09)
Type copyright, credits or license for more information.
IPython 2.0.0 -- An enhanced Interactive Python.
? - Introduction and overview of IPython's features.
%quickref - Quick reference.
help - Python's own help system.
object? - Details about 'object', use 'object??' for extra details.
In [1]: import numpy
In [2]: numpy.version.version
# My Question
What is going on there? Is this because I'm using Nix on a host OS, instead
of NixOS?
Or, am I just missing some package in the environment? In which case,
shouldn't Nix take care of this, and make sure all dependencies are
available?
One instance of a kind-of missing dependency is already the readline
module. If I don't mention it explicitly in the environment, python is
going to complain about it missing.
Thanks in advance for your help. And also thanks for doing this project in
the first place.
Best regards,
Andreas
[1]: https://nixos.org/nix/install
[2]:
https://nixos.org/wiki/Howto_develop_software_on_nixos#Obtaining_an_Environment_from_a_Custom_Definition
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] myEnvFun for IPython with Nix on openSuse host
Dear Luca,
Thanks for your response.
What commit of nixpkgs are you tracking?
How do I find that out?
I installed Nix using the install script, not the from source installation.
So, I never manually checked out any package list.
I ran the following commands. They didn't really do anything. In my
interpretation that means, that my Nix installation should already be
up-to-date. Correct?
$ nix-channel --list
nixpkgs http://nixos.org/channels/nixpkgs-unstable
$ nix-channel --update
downloading Nix expressions from `
http://releases.nixos.org/nixpkgs/nixpkgs-14.10pre44162.11ac14f/nixexprs.tar.xz'.
..
% Total% Received % Xferd Average Speed TimeTime Time
Current
Dload Upload Total SpentLeft
Speed
0 00 00 0 0 0 --:--:-- --:--:-- --:--:--
0
unpacking channels...
$ nix-env -u \*
$
Cheers.
On 1 June 2014 16:35, Luca Bruno lethalma...@gmail.com wrote:
What commit of nixpkgs are you tracking? Try upgrding your nixpkgs, that
should have been fixed a while ago.
On Sun, Jun 1, 2014 at 4:12 PM, Andreas Herrmann andreas...@gmx.ch
wrote:
Hi everyone,
I am very new to Nix, so let me apologize in advance if this ends up
being a stupid question.
# First, some back story:
The reason I got interested in Nix is that I am doing scientific
computing on a number of different machines with self-made simulation
software. Hence, I need to keep my development environments in sync on
these various machines. A tiresome task that, so far, I am performing
manually. I understand that Nix is promising to help in such situations.
So, I went ahead and installed Nix in single user mode on my laptop which
is running on openSuse 13.1. That is, I used the install script [1], and
installed Nix under /nix. On the wiki I found a section [2] about managing
development environments, which introduces a thing called myEnvFun. It
worked out for a simple environment to use numpy:
numpy = pkgs.myEnvFun {
name = numpy;
buildInputs = with pkgs; let p27=python27Packages; in [
stdenv python
p27.readline
p27.numpy
];
};
I've installed it using `nix-env -i env-numpy`, and loaded it using `
load-env-numpy`, and everything seems to work fine.
$ load-env-numpy
env-numpy loaded
$ which python
/nix/store/qikwsrbsyx59ygcmb36p1rgmfclgg882-python-2.7.6/bin/python
$ python
Python 2.7.6 (default, Apr 21 2014, 11:09:09)
[GCC 4.8.2] on linux2
Type help, copyright, credits or license for more information.
import numpy.version
numpy.version.version
'1.7.1'
However, I would prefer to do develop using the IPython shell. So, I
wrote another environment that encorporates ipython:
ipython = pkgs.myEnvFun {
name = ipython;
buildInputs = with pkgs; let p27=python27Packages; in [
stdenv python
p27.readline
p27.ipython
p27.numpy
];
};
Installing it works just fine. Loading the environment on the other hand
doesn't.
# The Problem
I get a huge amount errors about gdk-bug when I load the ipython
environment defined above:
$ load-env-ipython
mkdir: cannot create directory ‘/lib/gdk-pixbuf-loaders-2.0/’:
Permission denied
# repeated 61 times
mkdir: cannot create directory ‘/lib/gdk-pixbuf-loaders-2.0/’:
Permission denied
/nix/store/1j8nx7r6a7s76hj14b7lgqij7yddmvw2-gdk-pixbuf-2.30.6/nix-support/setup-hook:
line 11: /lib/gdk-pixbuf-loaders-2.0//loaders.cache: No such file or
directory
mkdir: cannot create directory ‘/lib/gdk-pixbuf-loaders-2.0/’:
Permission denied
# repeated 17 times
mkdir: cannot create directory ‘/lib/gdk-pixbuf-loaders-2.0/’:
Permission denied
env-ipython loaded
Depsite the error messages, the environment seems to work.
$ ipython
Python 2.7.6 (default, Apr 21 2014, 11:09:09)
Type copyright, credits or license for more information.
IPython 2.0.0 -- An enhanced Interactive Python.
? - Introduction and overview of IPython's features.
%quickref - Quick reference.
help - Python's own help system.
object? - Details about 'object', use 'object??' for extra details.
In [1]: import numpy
In [2]: numpy.version.version
# My Question
What is going on there? Is this because I'm using Nix on a host OS,
instead of NixOS?
Or, am I just missing some package in the environment? In which case,
shouldn't Nix take care of this, and make sure all dependencies are
available?
One instance of a kind-of missing dependency is already the readline
module. If I don't mention it explicitly in the environment, python is
going to complain about it missing.
Thanks in advance for your help. And also thanks for doing this project
in the first place.
Best regards,
Andreas
[1]: https://nixos.org/nix/install
[2]:
https://nixos.org/wiki
