Re: [Nix-dev] enableParallelbuilding for many packages

2012-10-26 Thread Vladimír Čunát
On 10/26/2012 12:36 AM, Marc Weber wrote: Buildfarm: You're wrong and you know it. Parallelizing nix builds is easy (scaling horizontally) - You can write your own Amazon buildfarm within one day which scales automatically probably. The problem is "time to get a new build" - and for that (securit

Re: [Nix-dev] enableParallelbuilding for many packages

2012-10-25 Thread jack-nix
On Wed, Oct 24, 2012 at 11:28:20PM +0200, Marc Weber wrote: > for each package if enableParallelbuilding is not set yet it will build > it three times: > 1st to build all dependencies (with enableParallelbuilding) > 2nd without enableParallelbuilding > 3rd with enableParallelbuilding > If all

Re: [Nix-dev] enableParallelbuilding for many packages

2012-10-25 Thread Marc Weber
Buildfarm: You're wrong and you know it. Parallelizing nix builds is easy (scaling horizontally) - You can write your own Amazon buildfarm within one day which scales automatically probably. The problem is "time to get a new build" - and for that (security updates etc) making each build go as fast

Re: [Nix-dev] enableParallelbuilding for many packages

2012-10-25 Thread Vladimír Čunát
Hi. On 10/24/2012 11:28 PM, Marc Weber wrote: People who think accuracy is most important will disable this feature anyway - but some like me sometimes just have to do rebuilds .. and they may just work more often than not. For the regular build farm work the total speedup will probably be ne

Re: [Nix-dev] enableParallelbuilding for many packages

2012-10-25 Thread Ludovic Courtès
Hi, Marc Weber skribis: >> Presumably, yes. > wireless-tools-29 was only built once, right? Well, I tried to build everything just once. So when one package failed to build with -jN, I didn’t try another time to see if it would magically build; knowing that a build succeeds X % of the time wou

Re: [Nix-dev] enableParallelbuilding for many packages

2012-10-24 Thread Marc Weber
> Presumably, yes. wireless-tools-29 was only built once, right? I want to have at least some safety and make race conditions less likely by building each package at least twice. Which in turn means that we are like to require this: enableParallelbuilding = 4; # don't set N to a higher value tha

Re: [Nix-dev] enableParallelbuilding for many packages

2012-10-24 Thread Ludovic Courtès
Marc Weber skribis: > Using your results would speed up my testing dramatically. > So I could use all names which succeeded for at least -j2 and -j4? Presumably, yes. > Eg how to read the wget results? They are only available for -jN N=2, > but not for greater N That means that either wget or

Re: [Nix-dev] enableParallelbuilding for many packages

2012-10-24 Thread Marc Weber
> http://hubble.gforge.inria.fr/parallel-builds.html > http://hubble.gforge.inria.fr/parallel-build-details.html Mmh. Why didn't we turn your results into action? Using your results would speed up my testing dramatically. So I could use all names which succeeded for at least -j2 and -j4? Eg h

Re: [Nix-dev] enableParallelbuilding for many packages

2012-10-24 Thread Ludovic Courtès
Hi, Marc Weber skribis: > Some results are already available here: > http://mawercer.de/cache FWIW, I had collected build speedups at: http://hubble.gforge.inria.fr/parallel-builds.html http://hubble.gforge.inria.fr/parallel-build-details.html Thanks, Ludo’. _

[Nix-dev] enableParallelbuilding for many packages

2012-10-24 Thread Marc Weber
I've written a small quick and dirty ruby script: http://mawercer.de/run-tests.rb for each package if enableParallelbuilding is not set yet it will build it three times: 1st to build all dependencies (with enableParallelbuilding) 2nd without enableParallelbuilding 3rd with enableParallelbuilding