On Sep 25, 2014 8:19 PM, "Peter Simons" <sim...@cryp.to> wrote: > > Hi Wout, > > > Another option would be to make Hydra super fast... What has been > > explored to optimize compile speeds? Using distcc, ccache, SSD, > > elastic scaling? > > Hydra is appears slow because "hydra-evaluator" is single-threaded. A > round-trip evaluating all jobsets on hydra.nixos.org takes almost a day. If > a commit comes in 10 minutes after 'master' was evaluated, then it takes ~24 > hours before the first build with that commit even *starts*. Once the > evaluator has put pending builds into the queue, however, you'll find that > the build slaves are super fast, actually. The throughput is great, but the > latency sucks.
Hmmm, I'm not sure I understand. Are you saying that build slaves are going idle because of build scheduling? > The other factor is that only a handful of people have administrator > privileges on the machines that run those builds, and those people are > volunteers with jobs, families, and a life. We've seen that 'master' can be > stuck for several days because of trivial system issues (disk space, > anyone?), but none of the people who could do something about it have time > to actually do it. I hereby volunteer. > > What if we had a security build fund that we could use to briefly run 500 > > machines to complete security builds? Would that allow 2-hour security > > rollouts? > > If we're looking only at Linux/x86_64, then 3-5 fast machines could easily > build all important packages in 'master' in approximately two hours. In > fact, hydra.nixos.org could do probably do that, too. The problem isn't the > build slaves -- the problem is the server that drives them. > > To improve the situation, we could: > > (1) Hack hydra-evaluator to be super efficient. > > (2) Rent some machine with lots of disk space, lots of memory, and a good > Internet connection to build Nixpkgs 'master', 'release-14.04', but > nothing else. Then have a team of 5-10 volunteers administer that > machine to guarantee responsiveness. Sounds good... > If we'd have some kind of "emergency response fund" that would allow us to > span build slaves in EC2, it would help matters greatly, too, but only if > there is someone who can quickly re-configure the main server to take > advantages of those slaves. Personally, I could easily spawn half a dozen > build slaves in EC2 and donate them to nixos.org for a day or two, but then > I'd have no way to configure hydra.nixos.org to use them! Sounds like a great problem to solve with NixOps! Wout.
_______________________________________________ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev