Re: [Nix-dev] Set up a Sufficiently Powerful Build Farm

[Charles Strahan]

This is great news :)

Rob, do you know how much of a difference the additional Mac Minis have
made?

Also, to which address can I ship a Mac Mini? If another Mini would make
a difference, I'd be more than happy to buy and ship one to the
foundation.

Charles

On Wed, Dec 23, 2015, at 12:12 PM, Rob Vermaas wrote:
> Hi,
> 
> On Tue, Dec 22, 2015 at 2:23 PM, Daniel Peebles 
> wrote:
> > I don't have a good sense of how many the foundation can keep, but for now
> > I'm not concerned about us having too many :) the more we get, the more
> > responsive we'll be.
> >
> > As it stands, I think we have two (oldish) Darwin cores on Hydra today, so
> > when my 4-core box arrives (assuming it works well) it should give us more
> > than 3x the current power :) I paid $589 for it and another $55 for shipping
> > from the US. Most of the ones for sale on eBay are the newer (but not
> > newest) generation and are thus a bit more expensive than the one I got.
> >
> > I think we'd be in a reasonably comfortable place if we could get 3 or 4
> > more of those boxes, but if we could get more I obviously wouldn't complain.
> >
> > I'd love for us to have Darwin be a nearly first-class citizen of nixpkgs,
> > but potential adopters (and thus contributors) are going to be turned off if
> > they need to recompile the world if they get anywhere near master.
> 
> Thanks to Dan for arranging an extra (powerful) Mac Mini and donating
> it to the Foundation!
> 
> Additionally, in the next few weeks, the NixOS Foundation will buy 2
> more Mac Mini's, which, together with Dan's Mac Mini and the existing
> one, should improve our Darwin build times significantly. We will look
> for further expansion of the Mac fleet once we get more information on
> the effects of the additional machines.
> 
> Cheers,
> Rob
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Set up a Sufficiently Powerful Build Farm

[Rob Vermaas]

Hi,

On Tue, Dec 22, 2015 at 2:23 PM, Daniel Peebles  wrote:
> I don't have a good sense of how many the foundation can keep, but for now
> I'm not concerned about us having too many :) the more we get, the more
> responsive we'll be.
>
> As it stands, I think we have two (oldish) Darwin cores on Hydra today, so
> when my 4-core box arrives (assuming it works well) it should give us more
> than 3x the current power :) I paid $589 for it and another $55 for shipping
> from the US. Most of the ones for sale on eBay are the newer (but not
> newest) generation and are thus a bit more expensive than the one I got.
>
> I think we'd be in a reasonably comfortable place if we could get 3 or 4
> more of those boxes, but if we could get more I obviously wouldn't complain.
>
> I'd love for us to have Darwin be a nearly first-class citizen of nixpkgs,
> but potential adopters (and thus contributors) are going to be turned off if
> they need to recompile the world if they get anywhere near master.

Thanks to Dan for arranging an extra (powerful) Mac Mini and donating
it to the Foundation!

Additionally, in the next few weeks, the NixOS Foundation will buy 2
more Mac Mini's, which, together with Dan's Mac Mini and the existing
one, should improve our Darwin build times significantly. We will look
for further expansion of the Mac fleet once we get more information on
the effects of the additional machines.

Cheers,
Rob
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Set up a Sufficiently Powerful Build Farm

[Christian Theune]

Hi,

another option that I didn’t mention but researched was looking for hosted Mac 
machines, but I find those offensively expensive (around 60€ per smallest mac 
mini per month). That means the price of a machine every 10 months.

Cheers,
Christian

> On 22 Dec 2015, at 10:27, Christian Theune  wrote:
> 
> Hi,
> 
>> On 29 Oct 2015, at 14:37, Eelco Dolstra > > wrote:
>> Thus the main problem right now is Darwin
>> builds, which are done on a single Mac mini…
> 
> This is actually also becoming a pain for me as even the channel branches 
> aren’t populated with the most basic packages and I keep compiling the world 
> for even the most simple package update on OS X.
> 
> What’s the biggest blocker here? Is it money for another machine? Is it 
> someone taking care of it? Is it placing them somewhere with a reasonable 
> uplink?
> 
> I could put one or more mac minis in our office on a reasonable big pipe 
> (100mbit fibre up/down). Question is whether we could fund them somehow. 
> Would a single additional machine help or do we need multiple? (For 
> anchoring: the app store prices the smallest one at 569€ and the question is: 
> would we rather have multiple small ones or a bigger one? Or would a bigger 
> investment into a Mac Pro make sense?
> 
> But before going into details: what’s the biggest blocker at the moment?
> 
> Christian
> 
> --
> Christian Theune · c...@flyingcircus.io  · +49 
> 345 219401 0
> Flying Circus Internet Operations GmbH · http://flyingcircus.io 
> 
> Forsterstraße 29 · 06112 Halle (Saale) · Deutschland
> HR Stendal HRB 21169 · Geschäftsführer: Christian. Theune, Christian. 
> Zagrodnick
> 
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev

--
Christian Theune · c...@flyingcircus.io · +49 345 219401 0
Flying Circus Internet Operations GmbH · http://flyingcircus.io
Forsterstraße 29 · 06112 Halle (Saale) · Deutschland
HR Stendal HRB 21169 · Geschäftsführer: Christian. Theune, Christian. Zagrodnick



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Set up a Sufficiently Powerful Build Farm

[Christian Theune]

Hi,

> On 29 Oct 2015, at 14:37, Eelco Dolstra  wrote:
> Thus the main problem right now is Darwin
> builds, which are done on a single Mac mini…

This is actually also becoming a pain for me as even the channel branches 
aren’t populated with the most basic packages and I keep compiling the world 
for even the most simple package update on OS X.

What’s the biggest blocker here? Is it money for another machine? Is it someone 
taking care of it? Is it placing them somewhere with a reasonable uplink?

I could put one or more mac minis in our office on a reasonable big pipe 
(100mbit fibre up/down). Question is whether we could fund them somehow. Would 
a single additional machine help or do we need multiple? (For anchoring: the 
app store prices the smallest one at 569€ and the question is: would we rather 
have multiple small ones or a bigger one? Or would a bigger investment into a 
Mac Pro make sense?

But before going into details: what’s the biggest blocker at the moment?

Christian

--
Christian Theune · c...@flyingcircus.io · +49 345 219401 0
Flying Circus Internet Operations GmbH · http://flyingcircus.io
Forsterstraße 29 · 06112 Halle (Saale) · Deutschland
HR Stendal HRB 21169 · Geschäftsführer: Christian. Theune, Christian. Zagrodnick



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Set up a Sufficiently Powerful Build Farm

[Profpatsch]

On 15-12-22 10:27am, Christian Theune wrote:
> I could put one or more mac minis in our office on a reasonable big pipe 
> (100mbit fibre up/down). Question is whether we could fund them somehow. 
> Would a single additional machine help or do we need multiple? (For 
> anchoring: the app store prices the smallest one at 569€ and the question is: 
> would we rather have multiple small ones or a bigger one? Or would a bigger 
> investment into a Mac Pro make sense?

Not to be the downer here, but that’s what you signed up for when you bought 
Mac.
Now don’t complain and hand over the money, as you promised.

I’m still astounded at whatever people expect.

-- 
Written on a $300 T400


signature.asc
Description: PGP signature
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Set up a Sufficiently Powerful Build Farm

[Domen Kožar]

Please, let's stay constructive. We can't change the hardware industry, but
only be a complement to it.

On Tue, Dec 22, 2015 at 12:47 PM, Profpatsch  wrote:

> On 15-12-22 10:27am, Christian Theune wrote:
> > I could put one or more mac minis in our office on a reasonable big pipe
> (100mbit fibre up/down). Question is whether we could fund them somehow.
> Would a single additional machine help or do we need multiple? (For
> anchoring: the app store prices the smallest one at 569€ and the question
> is: would we rather have multiple small ones or a bigger one? Or would a
> bigger investment into a Mac Pro make sense?
>
> Not to be the downer here, but that’s what you signed up for when you
> bought Mac.
> Now don’t complain and hand over the money, as you promised.
>
> I’m still astounded at whatever people expect.
>
> --
> Written on a $300 T400
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Set up a Sufficiently Powerful Build Farm

[Christian Theune]

Hi,

> On 22 Dec 2015, at 14:57, Daniel Peebles  wrote:
> 
> I spoke with Rob Vermaas about donating Darwin machines and we decided that 
> the hosted Mac options were indeed too expensive. As far as I could tell, the 
> best Mac compute power for the money is a Mac Mini from the previous 
> generation (or the one before it) since they had 4-core i7 CPUs back then 
> (the more recent ones only have 2-core CPUs and often i5s unless you spend 
> lots of money).
> 
> I bought one of those older Mac Minis off eBay as a donation to the NixOS 
> foundation, and once it ships to them they will manage the machine as part of 
> the main Hydra cluster.
> 
> It would sometimes be more appealing to host the machines ourselves, but 
> unfortunately there are all sorts of trust issues there which we haven't yet 
> solved, so I think the best bet for now if you want Darwin Hydra to get 
> stronger is either to donate to the foundation so they can buy more (Darwin) 
> build boxes, or to donate hardware directly as I did.

Thanks. If that’s feasible and works for the guys maintaining the hardware at 
the farm then I wonder: do we have an estimate how many machines would make an 
impact or should we try adding a bunch more? Or just wait for yours to arrive?

How much did you pay for the mac mini you managed to get?

Christian

--
Christian Theune · c...@flyingcircus.io · +49 345 219401 0
Flying Circus Internet Operations GmbH · http://flyingcircus.io
Forsterstraße 29 · 06112 Halle (Saale) · Deutschland
HR Stendal HRB 21169 · Geschäftsführer: Christian. Theune, Christian. Zagrodnick



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Set up a Sufficiently Powerful Build Farm

[Christian Theune]

Hi,

> On 22 Dec 2015, at 12:47, Profpatsch  wrote:
> 
> On 15-12-22 10:27am, Christian Theune wrote:
>> I could put one or more mac minis in our office on a reasonable big pipe 
>> (100mbit fibre up/down). Question is whether we could fund them somehow. 
>> Would a single additional machine help or do we need multiple? (For 
>> anchoring: the app store prices the smallest one at 569€ and the question 
>> is: would we rather have multiple small ones or a bigger one? Or would a 
>> bigger investment into a Mac Pro make sense?
> 
> Not to be the downer here, but that’s what you signed up for when you bought 
> Mac.
> Now don’t complain and hand over the money, as you promised.
> 
> I’m still astounded at whatever people expect.

What, who, where? I didn’t mean to imply expectation from my side getting 
anything for free.

My impression was there are multiple (don’t know how many) people interested in 
central builds for OS X and thus there’s a subset of the community interested 
in maintaining and investing into the OS X build farm. I’m trying to sort out 
what our options are and whether and where I can contribute. Not sure which 
wire I tripped for you. :)

Cheers,
Christian

--
Christian Theune · c...@flyingcircus.io · +49 345 219401 0
Flying Circus Internet Operations GmbH · http://flyingcircus.io
Forsterstraße 29 · 06112 Halle (Saale) · Deutschland
HR Stendal HRB 21169 · Geschäftsführer: Christian. Theune, Christian. Zagrodnick



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Set up a Sufficiently Powerful Build Farm

[Daniel Peebles]

I don't have a good sense of how many the foundation can keep, but for now
I'm not concerned about us having too many :) the more we get, the more
responsive we'll be.

As it stands, I think we have two (oldish) Darwin cores on Hydra today, so
when my 4-core box  arrives (assuming
it works well) it should give us more than 3x the current power :) I paid
$589 for it and another $55 for shipping from the US. Most of the ones for
sale on eBay are the newer (but not newest) generation and are thus a bit
more expensive than the one I got.

I think we'd be in a reasonably comfortable place if we could get 3 or 4
more of those boxes, but if we could get more I obviously wouldn't complain.

I'd love for us to have Darwin be a nearly first-class citizen of nixpkgs,
but potential adopters (and thus contributors) are going to be turned off
if they need to recompile the world if they get anywhere near master.




On Tue, Dec 22, 2015 at 9:16 AM, Christian Theune 
wrote:

> Hi,
>
> On 22 Dec 2015, at 14:57, Daniel Peebles  wrote:
>
> I spoke with Rob Vermaas about donating Darwin machines and we decided
> that the hosted Mac options were indeed too expensive. As far as I could
> tell, the best Mac compute power for the money is a Mac Mini from the
> previous generation (or the one before it) since they had 4-core i7 CPUs
> back then (the more recent ones only have 2-core CPUs and often i5s unless
> you spend lots of money).
>
> I bought one of those older Mac Minis off eBay as a donation to the NixOS
> foundation, and once it ships to them they will manage the machine as part
> of the main Hydra cluster.
>
> It would sometimes be more appealing to host the machines ourselves, but
> unfortunately there are all sorts of trust issues there which we haven't
> yet solved, so I think the best bet for now if you want Darwin Hydra to get
> stronger is either to donate to the foundation so they can buy more
> (Darwin) build boxes, or to donate hardware directly as I did.
>
>
> Thanks. If that’s feasible and works for the guys maintaining the hardware
> at the farm then I wonder: do we have an estimate how many machines would
> make an impact or should we try adding a bunch more? Or just wait for yours
> to arrive?
>
> How much did you pay for the mac mini you managed to get?
>
> Christian
>
> --
> Christian Theune · c...@flyingcircus.io · +49 345 219401 0
> Flying Circus Internet Operations GmbH · http://flyingcircus.io
> Forsterstraße 29 · 06112 Halle (Saale) · Deutschland
> HR Stendal HRB 21169 · Geschäftsführer: Christian. Theune, Christian.
> Zagrodnick
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Set up a Sufficiently Powerful Build Farm

[Chris Forno]

Peter Simons  cryp.to> writes:

> Anyhow, that's just a rough estimate. I don't know, really, what an ideal
> hardware / service platform for running such a virtual service would be. It
> would be great if a resident virtual server / NAS / system management guru
> could chime in with suggestions; I'm sure the NixOS crowd has people who know
> that kind of stuff and who can design the infrastructure for such a build
> farm.

I'd be happy to help, as I think I can use the experience to figure out
archiving build
sources ().
Ideally we can find a way to not only allow a user to populate a Nix store on
their machine with all sources for a given nixpkgs checkout but also to ship
the sources of all successful hydra builds to some long-term storage.

It would be interesting to also measure/project the storage required to
archive all builds produced by hydra across time.

___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Set up a Sufficiently Powerful Build Farm

[Peter Simons]

Hi Eelco,

 >>> Just to be clear: you mean it compresses *to* 8%, or *by* 8%?
 >>
 >> By 8%.
 >
 > Hm, that's surprisingly low.

it turns out that nix-push compressed to ~8%, actually. I did not read
nix-push's out output correctly.

I repeated the experiment a moment ago, and a closure of all active
builds in 'haskellPackages' comes out weighing 2,212 MiByte (instead of
the uncompressed 31 GiByte) on x86_64-linux. I suppose this means the
build products from *all* Haskell package sets for *all* platforms would
take up approximately 100 GiByte in compressed form.

Best regards,
Peter

___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Set up a Sufficiently Powerful Build Farm

[Peter Simons]

Hi Eelco,

 > Regarding hydra.nixos.org disk space, I hope to address this by
 > having hydra-queue-runner store build results directly in an S3
 > bucket (i.e.cache.nixos.org), rather than in the local Nix store.

yes, that sounds like a good solution. It would be great to have that
feature available in hydra.nixos.org. Especially, if the change also
increases the performance of the evaluator.


 > Packages would appear in cache.nixos.org as soon as they have been
 > built, rather than when the channel mirror script runs.

Do you have any idea how to deal with the fact that some jobsets aren't
necessarily supposed to be available from cache.nixos.org, like
nixpkgs/multiple-outputs, because they're used for CI-testing work in
progress rather than for producing release binaries of Nixpkgs? There
seems to be an interesting problem insofar as that we don't know whether
a build run as part of "staging" is going to end up in the "unstable"
channel or not at the time when we run it. If the evaluation as a whole
is good, we merge "staging" to "master" and the build becomes a part of
Nixpkgs, but it might also happen that we have another stdenv change in
"staging" before such a merge occurs, and then the binaries produced by
that builds turn out to be irrelevant.


 > Do you have data on how well those packages compress? Cache.nixos.org
 > uses xz compression, so if hydra-queue-runner compressed build
 > results before uploading to S3, it would presumably need a lot less
 > than 1 TB.

I've nix-push'ed some 1,500 packages for testing purposes, and the tool
reports a total compression ratio of ~8%. That number is somewhat
inaccurate because nix-push publishes the *closure* of each store path I
give it -- so the compression ratio is computed over more than just the
Haskell packages, but still it feels like a reasonable result.

Generally speaking, the 1TB estimate is probably on the high end of the
scale, because it's based on the assumption that the size of the Haskell
package set for all platforms equals 3 times the size of that from
Linux/x86_64. In practice, however, i686 binaries will be smaller than
those from 86_64, and we also run far less builds on Darwin than on
Linux.

Anyway, I've also taken a moment to look at the build times of the
Haskell package set. On a moderately recent Linux/x86_64 server, the
build run-times (in minutes) are distributed as follows (based on
259,455 samples, not including "ghc" builds):

   Min.  1st Qu.   Median Mean  3rd Qu. Max.
0.09744  0.22790  0.37780  0.92970  0.75990 81.83000

The most expensive builds in terms of average run-time are:

  job builds   runtime medianmean sd
 1:  metadata 14 1145.6167 74.77 81.82976  23.462635
 2:poker-eval 35 2136.6000 48.50 61.04571  29.052927
 3:unicode-properties 13  458.9667 34.05 35.30513   8.226384
 4:   FpMLv53 19  652.8000 30.416667 34.35789  15.851984
 5:   texmath 91 3106.6000 21.55 34.13846  50.151076
 6:  scholdoc-texmath 77 2437.1500 32.00 31.65130   6.629542
 7:  amazonka-ec2109 3423.9500 28.63 31.41239  15.110833
 8: csound-expression-opcodes 27  846.7000  3.28 31.35926 142.592864
 9:  Agda 93 2860.5667 25.55 30.75878  14.754584
10: idris128 3269.7667 23.491667 25.54505   8.495109
11:accelerate-fourier 15  373.9167 31.18 24.92778  19.153399
12: uhc-light 46 1110.5333 22.95 24.14203   8.405919
13:   RSA 81 1940.1333  1.60 23.95226  56.325163
14:java-character 16  362.5500 23.70 22.65938   3.750131
15:  encoding 33  689.6333 17.83 20.89798   9.091368
16:  lens117 2421.4667 18.70 20.69630   8.762991
17: unicode-names 12  238.3833 20.058333 19.86528   1.812477
18:  hssqlppp 55 1019.6667 15.57 18.53939   6.277109
19:   uhc 21  369.0833 17.10 17.57540   5.794108
20:gl 74 1287.9167 17.875000 17.40428   5.709247
21:open-symbology 19  322.2667 16.98 16.96140   2.655905
22:  graphviz 53  894.6333 13.316667 16.87987  16.741832
23:  riak 47  782.3667  0.80 16.64610 108.513826
24:  haskell-src-exts 51  826.7000 14.53 16.20980   7.695385
25:   gtk 89 1408.8167 16.316667 15.82940   4.134229
26: git-annex510 7850.3833 13.891667 15.39291   6.257803
27:yi138 2101.7833 15.07 15.23031   3.314749
28:GenussFold 12  179.2333 14.941667 14.93611   4.535822
29:  midi 39  579.5667  2.97 14.86068  71.999306
30:  gtk3 89 1260.9667 

Re: [Nix-dev] Set up a Sufficiently Powerful Build Farm

[Eelco Dolstra]

Hi Peter,

On 04/11/15 14:52, Peter Simons wrote:

>  > Packages would appear in cache.nixos.org as soon as they have been
>  > built, rather than when the channel mirror script runs.
> 
> Do you have any idea how to deal with the fact that some jobsets aren't
> necessarily supposed to be available from cache.nixos.org, like
> nixpkgs/multiple-outputs, because they're used for CI-testing work in
> progress rather than for producing release binaries of Nixpkgs?

Everything would be stored in the binary cache right away, even packages that
don't end up in a NixOS/Nixpkgs channel. The main consequence is that we'd have
to run the binary cache garbage collection script [1] more frequently (currently
I manually run it once or twice a year).

[1]
https://github.com/NixOS/nixos-channel-scripts/blob/master/find-binary-cache-garbage.pl

>  > Do you have data on how well those packages compress? Cache.nixos.org
>  > uses xz compression, so if hydra-queue-runner compressed build
>  > results before uploading to S3, it would presumably need a lot less
>  > than 1 TB.
> 
> I've nix-push'ed some 1,500 packages for testing purposes, and the tool
> reports a total compression ratio of ~8%. 

Thanks!

Just to be clear: you mean it compresses *to* 8%, or *by* 8%?

-- 
Eelco Dolstra | LogicBlox, Inc. | http://nixos.org/~eelco/
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Set up a Sufficiently Powerful Build Farm

[Eelco Dolstra]

Hi,

On 04/11/15 17:28, Peter Simons wrote:
> Eelco Dolstra writes:
> 
>  > Just to be clear: you mean it compresses *to* 8%, or *by* 8%?
> 
> By 8%. "To" would be nice, though, wouldn't it? :-)

Hm, that's surprisingly low. (OTOH, compressing *to* 8% would be surprisingly
high...) For comparison, the store paths in my system closure obtained from the
binary cache compress to 25.3% (from 2882 MiB to 729 MiB).

-- 
Eelco Dolstra | LogicBlox, Inc. | http://nixos.org/~eelco/
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

[Nix-dev] Set up a Sufficiently Powerful Build Farm

[Peter Simons]

The Problem
---

hydra.nixos.org compiles and provides binaries only for the "haskellPackages"
package set. The build farm compiles none of our LTS Haskell package sets,
which means that users of "haskell.packages.lts-x_y" cannot get any
pre-compiled binaries. It also means that those builds aren't verified, i.e. we
won't notice when changes to Nixpkgs break builds in those package sets.

Furthermore, we have no pre-compiled binaries with profiling support [1] for
any of our package sets.

The Situation Today
---

We have 66 active package sets that define the following number of active
builds per platform:

pkgset builds
 1:ghc6123   5173
 2: ghc704   5182
 3:ghc7102   5189
 4: ghc722   5183
 5: ghc742   5183
 6: ghc763   5182
 7: ghc783   5173
 8: ghc784   5173
 9:ghcHEAD   5188
10: ghcNokinds   5188
11:  ghcjs   5172
12:lts-0_0795
13:lts-0_1795
14:lts-0_2795
15:lts-0_3795
16:lts-0_4795
17:lts-0_5795
18:lts-0_6795
19:lts-0_7795
20:lts-1_0827
21:lts-1_1827
22:   lts-1_10828
23:   lts-1_11829
24:   lts-1_12829
25:   lts-1_13829
26:   lts-1_14830
27:   lts-1_15831
28:lts-1_2828
29:lts-1_4828
30:lts-1_5828
31:lts-1_7828
32:lts-1_8828
33:lts-1_9827
34:lts-2_0   1019
35:lts-2_1   1019
36:   lts-2_10   1023
37:   lts-2_11   1023
38:   lts-2_12   1023
39:   lts-2_13   1022
40:   lts-2_14   1022
41:   lts-2_15   1022
42:   lts-2_16   1022
43:   lts-2_17   1023
44:   lts-2_18   1022
45:   lts-2_19   1022
46:lts-2_2   1018
47:   lts-2_20   1024
48:   lts-2_21   1023
49:   lts-2_22   1023
50:lts-2_3   1018
51:lts-2_4   1018
52:lts-2_5   1018
53:lts-2_6   1017
54:lts-2_7   1017
55:lts-2_8   1023
56:lts-2_9   1023
57:lts-3_0   1322
58:lts-3_1   1322
59:lts-3_2   1321
60:lts-3_3   1321
61:lts-3_4   1321
62:lts-3_5   1322
63:lts-3_6   1321
64:lts-3_7   1323
65:lts-3_8   1323
66:lts-3_9   1324
pkgset builds

That gives a total of 111,647 active builds, many of which are identical. All
package sets combined define 77,445 distinct store paths, i.e. some 34,202
builds are shared across package sets.

Now, hydra.nixos.org compiles only "haskellPackages" at the moment. Out of a
total of 46,862 builds in trunk [2], 15,446 (33%) come from the Haskell package
set. If we'd enable every Haskell package set on Linux/i686, Linux/x86_64, and
Darwin/x86_64, then we'd have a total of 263,751 builds -- 5.6 times as much as
before --, and 88% of all builds would be related to Haskell.

A complete build of the active derivations in "haskellPackages" takes up
approx. 27 GByte of disk space per platform. That gives about 80 GByte for all
of our 3 active platforms. How would that number develop if we'd enable
everything? The store path sizes in MByte are distributed as follows (based on
7,620 samples excluding "ghc"):

Minimum  1st Quart.  Median Mean   3rd Quart.  Maximum
0.0169   0.3557  0.9497   4.6300   3.0640  678.9000

Multiplying the average store path size by the number of distinct store paths
tells us that storing *everything* requires approx. 360 GByte per platform.
With 3 active platforms, we'd need about 1 TByte of disk space for one complete
set of Haskell packages.

Now, we might be able to reduce that number by disabling some particularly
large builds. The store path size distribution is skewed to the left, i.e.
towards smaller builds. Approximately 82% of all store paths are actually
smaller than the numerical average. Our top-20 biggest Haskell builds are:

pkg  size
 1: ghc 895.5
 2:metadata 678.9
 3:   uhc-light 253.5
 4:   OpenGLRaw 249.3
 5: FpMLv53 229.5
 6:amazonka-ec2 214.8
 7:Agda 212.6
 8: xhb 189.0
 9:  unicode-properties 175.0
10:scholdoc-texmath 165.0
11:   idris 137.7
12:  gf 126.4
13:  pandoc 124.8
14:   unicode-names 118.4
15:  wxcore 113.8
16:  java-character 112.1
17: hat 111.1
18: texmath 109.6
19:  open-symbology 107.6
20: turkish-deasciifier 104.0

If 

Re: [Nix-dev] Set up a Sufficiently Powerful Build Farm

[Eelco Dolstra]

Hi,

On 29/10/15 13:38, Peter Simons wrote:

> Multiplying the average store path size by the number of distinct store paths
> tells us that storing *everything* requires approx. 360 GByte per platform.
> With 3 active platforms, we'd need about 1 TByte of disk space for one 
> complete
> set of Haskell packages.

Do you have data on how well those packages compress? Cache.nixos.org uses xz
compression, so if hydra-queue-runner compressed build results before uploading
to S3, it would presumably need a lot less than 1 TB.

-- 
Eelco Dolstra | LogicBlox, Inc. | http://nixos.org/~eelco/
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Set up a Sufficiently Powerful Build Farm

[Eelco Dolstra]

Hi Peter,

On 29/10/15 13:38, Peter Simons wrote:



Regarding hydra.nixos.org disk space, I hope to address this by having
hydra-queue-runner store build results directly in an S3 bucket (i.e.
cache.nixos.org), rather than in the local Nix store. (That would be part 2 of
the queue runner overhaul - part 1 was having a single-process queue runner that
loads the dependency graphs of all builds in the queue and can thus schedule
builds more efficiently.)

This would have a few advantages:

* Unlimited disk space.

* Remove the Nix store on the central machine as a bottleneck. Currently copying
store paths to/from the build slaves takes significantly more time than the
actual builds.

* Shorter latency - packages would appear in cache.nixos.org as soon as they
have been built, rather than when the channel mirror script runs.

The Nix store on the central machine would then only be used to communicate .drv
files between hydra-evaluator and hydra-queue-runner. But this shouldn't be much
data, so it would enable using a fast SSD drive for the Nix store.

Another recent improvement is hydra-provisioner, which is a script that fires up
EC2 spot instances dynamically based on the Hydra queue. This allows us to burn
through Linux builds pretty quickly. Thus the main problem right now is Darwin
builds, which are done on a single Mac mini...

> It's unclear whether the Hydra software would cope with 66 package sets with
> some 111,000 derivations in them that need to be evaluated, say, once an hour.
> Hydra has undergone some architectural changes recently that might make such a
> load possible -- i.e. "hydra-evaluator" is more efficient than it used to be
> --, but I don't have any reliable data concerning the performance of the
> process, so I cannot say what is possible and what is not.

Currently the main cause of evaluator slowness on hydra.nixos.org is due to the
high I/O load and the disk not being very fast in general. If the store, Nix
database and PostgreSQL database were on an SSD drive, I expect the evaluator to
be much faster.

-- 
Eelco Dolstra | LogicBlox, Inc. | http://nixos.org/~eelco/
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev