On Fri, Oct 10, 2014 at 4:34 PM, Andreas Herrmann <andreas...@gmx.ch> wrote:
> Thank you for the detailed answer. > > On Friday 10 October 2014 15:32:52 Wout Mertens wrote: > > I think you could do this. You would set it up so the nix server does the > > compiles and the grid runs distcc. See the wiki, the raspberry pi page > has > > explanations about distcc. > Oh, I didn't know that this worked outside of NixOS. I just can't find any > details on how to integrate distcc with sge. Do you have any experience > with that? > No sorry, but I'd just install distcc as a daemon on all nodes or else use sge for the distribution somehow... pretend that they're super long running batch jobs... > > Note that only one node can write to nix store at the same time due to > the > > db. > That's interesting. The multi-user nix manual says that you can have > multiple build users on one machine. Why does that work on one machine, but > not spread over several machines? I would assume that you need to connect > to some kind of db daemon anyway? > it's because of sqlite, it is safe on a single machine but not on NFS... > Another option is to have private nix stores on all nodes but nfs mount > all > > of them under the remote stores directory. That way nix-store will fetch > > missing packages from the remotes and store them locally. At least, > that's > > my understanding. > I don't think I understand this. By private, do you mean one per user? And > how do we mount all of them under one directory, do you mean some layered > fs? > No I mean one per node, so on the local disk. 50GB should be ok I think, or a few GB if that's hard but then do nix-collect-garbage often. > > As for the Intel compiler, that could be a challenge, but right now we > have > > several gcc versions and clang, so it's not impossible. You can decide > on a > > per-package basis which compiler to use. > Yes, my main concern are things like license keys, or - even worse - > license servers. I don't know how to approach these things in nix. As long > as you can mix your nix environment with an external environment this is > not an issue. So, as a simple example, you could use nix to manage your > version of boost, but use an intel compiler that is installed outside of > nix for user code. > so that's a bit harder. If you need a license connection and for the rest the build is stateless, you can make that work I think. You'd probably need to cheat with a userhook for stdenv to get the license connection info in there though. > Not sure how mpi would influence state, can you elaborate? > I meant this on a per job basis. In the case of mpi jobs the sge will > decide which nodes your job will be distributed over. This information will > be passed by a mixture of config files and environment variables. Hence, > such jobs would not work inside a pure nix-shell, because the environment > needs to be passed through. I don't think it's a big issue. Just something > to keep in mind. > more userhooking :) See https://nixos.org/wiki/Raspberry_Pi#Implementing_distcc Cheers, Wout.
_______________________________________________ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev