Re: [Nix-dev] Set up a Sufficiently Powerful Build Farm

[Charles Strahan]

This is great news :)

Rob, do you know how much of a difference the additional Mac Minis have
made?

Also, to which address can I ship a Mac Mini? If another Mini would make
a difference, I'd be more than happy to buy and ship one to the
foundation.

Charles

On Wed, Dec 23, 2015, at 12:12 PM, Rob Vermaas wrote:
> Hi,
> 
> On Tue, Dec 22, 2015 at 2:23 PM, Daniel Peebles 
> wrote:
> > I don't have a good sense of how many the foundation can keep, but for now
> > I'm not concerned about us having too many :) the more we get, the more
> > responsive we'll be.
> >
> > As it stands, I think we have two (oldish) Darwin cores on Hydra today, so
> > when my 4-core box arrives (assuming it works well) it should give us more
> > than 3x the current power :) I paid $589 for it and another $55 for shipping
> > from the US. Most of the ones for sale on eBay are the newer (but not
> > newest) generation and are thus a bit more expensive than the one I got.
> >
> > I think we'd be in a reasonably comfortable place if we could get 3 or 4
> > more of those boxes, but if we could get more I obviously wouldn't complain.
> >
> > I'd love for us to have Darwin be a nearly first-class citizen of nixpkgs,
> > but potential adopters (and thus contributors) are going to be turned off if
> > they need to recompile the world if they get anywhere near master.
> 
> Thanks to Dan for arranging an extra (powerful) Mac Mini and donating
> it to the Foundation!
> 
> Additionally, in the next few weeks, the NixOS Foundation will buy 2
> more Mac Mini's, which, together with Dan's Mac Mini and the existing
> one, should improve our Darwin build times significantly. We will look
> for further expansion of the Mac fleet once we get more information on
> the effects of the additional machines.
> 
> Cheers,
> Rob
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Set up a Sufficiently Powerful Build Farm

[Rob Vermaas]

Hi,

On Tue, Dec 22, 2015 at 2:23 PM, Daniel Peebles  wrote:
> I don't have a good sense of how many the foundation can keep, but for now
> I'm not concerned about us having too many :) the more we get, the more
> responsive we'll be.
>
> As it stands, I think we have two (oldish) Darwin cores on Hydra today, so
> when my 4-core box arrives (assuming it works well) it should give us more
> than 3x the current power :) I paid $589 for it and another $55 for shipping
> from the US. Most of the ones for sale on eBay are the newer (but not
> newest) generation and are thus a bit more expensive than the one I got.
>
> I think we'd be in a reasonably comfortable place if we could get 3 or 4
> more of those boxes, but if we could get more I obviously wouldn't complain.
>
> I'd love for us to have Darwin be a nearly first-class citizen of nixpkgs,
> but potential adopters (and thus contributors) are going to be turned off if
> they need to recompile the world if they get anywhere near master.

Thanks to Dan for arranging an extra (powerful) Mac Mini and donating
it to the Foundation!

Additionally, in the next few weeks, the NixOS Foundation will buy 2
more Mac Mini's, which, together with Dan's Mac Mini and the existing
one, should improve our Darwin build times significantly. We will look
for further expansion of the Mac fleet once we get more information on
the effects of the additional machines.

Cheers,
Rob
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Set up a Sufficiently Powerful Build Farm

[Christian Theune]

Hi,

another option that I didn’t mention but researched was looking for hosted Mac 
machines, but I find those offensively expensive (around 60€ per smallest mac 
mini per month). That means the price of a machine every 10 months.

Cheers,
Christian

> On 22 Dec 2015, at 10:27, Christian Theune  wrote:
> 
> Hi,
> 
>> On 29 Oct 2015, at 14:37, Eelco Dolstra > > wrote:
>> Thus the main problem right now is Darwin
>> builds, which are done on a single Mac mini…
> 
> This is actually also becoming a pain for me as even the channel branches 
> aren’t populated with the most basic packages and I keep compiling the world 
> for even the most simple package update on OS X.
> 
> What’s the biggest blocker here? Is it money for another machine? Is it 
> someone taking care of it? Is it placing them somewhere with a reasonable 
> uplink?
> 
> I could put one or more mac minis in our office on a reasonable big pipe 
> (100mbit fibre up/down). Question is whether we could fund them somehow. 
> Would a single additional machine help or do we need multiple? (For 
> anchoring: the app store prices the smallest one at 569€ and the question is: 
> would we rather have multiple small ones or a bigger one? Or would a bigger 
> investment into a Mac Pro make sense?
> 
> But before going into details: what’s the biggest blocker at the moment?
> 
> Christian
> 
> --
> Christian Theune · c...@flyingcircus.io  · +49 
> 345 219401 0
> Flying Circus Internet Operations GmbH · http://flyingcircus.io 
> 
> Forsterstraße 29 · 06112 Halle (Saale) · Deutschland
> HR Stendal HRB 21169 · Geschäftsführer: Christian. Theune, Christian. 
> Zagrodnick
> 
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev

--
Christian Theune · c...@flyingcircus.io · +49 345 219401 0
Flying Circus Internet Operations GmbH · http://flyingcircus.io
Forsterstraße 29 · 06112 Halle (Saale) · Deutschland
HR Stendal HRB 21169 · Geschäftsführer: Christian. Theune, Christian. Zagrodnick



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Set up a Sufficiently Powerful Build Farm

[Christian Theune]

Hi,

> On 29 Oct 2015, at 14:37, Eelco Dolstra  wrote:
> Thus the main problem right now is Darwin
> builds, which are done on a single Mac mini…

This is actually also becoming a pain for me as even the channel branches 
aren’t populated with the most basic packages and I keep compiling the world 
for even the most simple package update on OS X.

What’s the biggest blocker here? Is it money for another machine? Is it someone 
taking care of it? Is it placing them somewhere with a reasonable uplink?

I could put one or more mac minis in our office on a reasonable big pipe 
(100mbit fibre up/down). Question is whether we could fund them somehow. Would 
a single additional machine help or do we need multiple? (For anchoring: the 
app store prices the smallest one at 569€ and the question is: would we rather 
have multiple small ones or a bigger one? Or would a bigger investment into a 
Mac Pro make sense?

But before going into details: what’s the biggest blocker at the moment?

Christian

--
Christian Theune · c...@flyingcircus.io · +49 345 219401 0
Flying Circus Internet Operations GmbH · http://flyingcircus.io
Forsterstraße 29 · 06112 Halle (Saale) · Deutschland
HR Stendal HRB 21169 · Geschäftsführer: Christian. Theune, Christian. Zagrodnick



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Set up a Sufficiently Powerful Build Farm

[Profpatsch]

On 15-12-22 10:27am, Christian Theune wrote:
> I could put one or more mac minis in our office on a reasonable big pipe 
> (100mbit fibre up/down). Question is whether we could fund them somehow. 
> Would a single additional machine help or do we need multiple? (For 
> anchoring: the app store prices the smallest one at 569€ and the question is: 
> would we rather have multiple small ones or a bigger one? Or would a bigger 
> investment into a Mac Pro make sense?

Not to be the downer here, but that’s what you signed up for when you bought 
Mac.
Now don’t complain and hand over the money, as you promised.

I’m still astounded at whatever people expect.

-- 
Written on a $300 T400


signature.asc
Description: PGP signature
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Set up a Sufficiently Powerful Build Farm

[Domen Kožar]

Please, let's stay constructive. We can't change the hardware industry, but
only be a complement to it.

On Tue, Dec 22, 2015 at 12:47 PM, Profpatsch  wrote:

> On 15-12-22 10:27am, Christian Theune wrote:
> > I could put one or more mac minis in our office on a reasonable big pipe
> (100mbit fibre up/down). Question is whether we could fund them somehow.
> Would a single additional machine help or do we need multiple? (For
> anchoring: the app store prices the smallest one at 569€ and the question
> is: would we rather have multiple small ones or a bigger one? Or would a
> bigger investment into a Mac Pro make sense?
>
> Not to be the downer here, but that’s what you signed up for when you
> bought Mac.
> Now don’t complain and hand over the money, as you promised.
>
> I’m still astounded at whatever people expect.
>
> --
> Written on a $300 T400
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Set up a Sufficiently Powerful Build Farm

[Christian Theune]

Hi,

> On 22 Dec 2015, at 14:57, Daniel Peebles  wrote:
> 
> I spoke with Rob Vermaas about donating Darwin machines and we decided that 
> the hosted Mac options were indeed too expensive. As far as I could tell, the 
> best Mac compute power for the money is a Mac Mini from the previous 
> generation (or the one before it) since they had 4-core i7 CPUs back then 
> (the more recent ones only have 2-core CPUs and often i5s unless you spend 
> lots of money).
> 
> I bought one of those older Mac Minis off eBay as a donation to the NixOS 
> foundation, and once it ships to them they will manage the machine as part of 
> the main Hydra cluster.
> 
> It would sometimes be more appealing to host the machines ourselves, but 
> unfortunately there are all sorts of trust issues there which we haven't yet 
> solved, so I think the best bet for now if you want Darwin Hydra to get 
> stronger is either to donate to the foundation so they can buy more (Darwin) 
> build boxes, or to donate hardware directly as I did.

Thanks. If that’s feasible and works for the guys maintaining the hardware at 
the farm then I wonder: do we have an estimate how many machines would make an 
impact or should we try adding a bunch more? Or just wait for yours to arrive?

How much did you pay for the mac mini you managed to get?

Christian

--
Christian Theune · c...@flyingcircus.io · +49 345 219401 0
Flying Circus Internet Operations GmbH · http://flyingcircus.io
Forsterstraße 29 · 06112 Halle (Saale) · Deutschland
HR Stendal HRB 21169 · Geschäftsführer: Christian. Theune, Christian. Zagrodnick



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Set up a Sufficiently Powerful Build Farm

[Christian Theune]

Hi,

> On 22 Dec 2015, at 12:47, Profpatsch  wrote:
> 
> On 15-12-22 10:27am, Christian Theune wrote:
>> I could put one or more mac minis in our office on a reasonable big pipe 
>> (100mbit fibre up/down). Question is whether we could fund them somehow. 
>> Would a single additional machine help or do we need multiple? (For 
>> anchoring: the app store prices the smallest one at 569€ and the question 
>> is: would we rather have multiple small ones or a bigger one? Or would a 
>> bigger investment into a Mac Pro make sense?
> 
> Not to be the downer here, but that’s what you signed up for when you bought 
> Mac.
> Now don’t complain and hand over the money, as you promised.
> 
> I’m still astounded at whatever people expect.

What, who, where? I didn’t mean to imply expectation from my side getting 
anything for free.

My impression was there are multiple (don’t know how many) people interested in 
central builds for OS X and thus there’s a subset of the community interested 
in maintaining and investing into the OS X build farm. I’m trying to sort out 
what our options are and whether and where I can contribute. Not sure which 
wire I tripped for you. :)

Cheers,
Christian

--
Christian Theune · c...@flyingcircus.io · +49 345 219401 0
Flying Circus Internet Operations GmbH · http://flyingcircus.io
Forsterstraße 29 · 06112 Halle (Saale) · Deutschland
HR Stendal HRB 21169 · Geschäftsführer: Christian. Theune, Christian. Zagrodnick



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Set up a Sufficiently Powerful Build Farm

[Daniel Peebles]

I don't have a good sense of how many the foundation can keep, but for now
I'm not concerned about us having too many :) the more we get, the more
responsive we'll be.

As it stands, I think we have two (oldish) Darwin cores on Hydra today, so
when my 4-core box  arrives (assuming
it works well) it should give us more than 3x the current power :) I paid
$589 for it and another $55 for shipping from the US. Most of the ones for
sale on eBay are the newer (but not newest) generation and are thus a bit
more expensive than the one I got.

I think we'd be in a reasonably comfortable place if we could get 3 or 4
more of those boxes, but if we could get more I obviously wouldn't complain.

I'd love for us to have Darwin be a nearly first-class citizen of nixpkgs,
but potential adopters (and thus contributors) are going to be turned off
if they need to recompile the world if they get anywhere near master.




On Tue, Dec 22, 2015 at 9:16 AM, Christian Theune 
wrote:

> Hi,
>
> On 22 Dec 2015, at 14:57, Daniel Peebles  wrote:
>
> I spoke with Rob Vermaas about donating Darwin machines and we decided
> that the hosted Mac options were indeed too expensive. As far as I could
> tell, the best Mac compute power for the money is a Mac Mini from the
> previous generation (or the one before it) since they had 4-core i7 CPUs
> back then (the more recent ones only have 2-core CPUs and often i5s unless
> you spend lots of money).
>
> I bought one of those older Mac Minis off eBay as a donation to the NixOS
> foundation, and once it ships to them they will manage the machine as part
> of the main Hydra cluster.
>
> It would sometimes be more appealing to host the machines ourselves, but
> unfortunately there are all sorts of trust issues there which we haven't
> yet solved, so I think the best bet for now if you want Darwin Hydra to get
> stronger is either to donate to the foundation so they can buy more
> (Darwin) build boxes, or to donate hardware directly as I did.
>
>
> Thanks. If that’s feasible and works for the guys maintaining the hardware
> at the farm then I wonder: do we have an estimate how many machines would
> make an impact or should we try adding a bunch more? Or just wait for yours
> to arrive?
>
> How much did you pay for the mac mini you managed to get?
>
> Christian
>
> --
> Christian Theune · c...@flyingcircus.io · +49 345 219401 0
> Flying Circus Internet Operations GmbH · http://flyingcircus.io
> Forsterstraße 29 · 06112 Halle (Saale) · Deutschland
> HR Stendal HRB 21169 · Geschäftsführer: Christian. Theune, Christian.
> Zagrodnick
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Set up a Sufficiently Powerful Build Farm

[Chris Forno]

Peter Simons  cryp.to> writes:

> Anyhow, that's just a rough estimate. I don't know, really, what an ideal
> hardware / service platform for running such a virtual service would be. It
> would be great if a resident virtual server / NAS / system management guru
> could chime in with suggestions; I'm sure the NixOS crowd has people who know
> that kind of stuff and who can design the infrastructure for such a build
> farm.

I'd be happy to help, as I think I can use the experience to figure out
archiving build
sources ().
Ideally we can find a way to not only allow a user to populate a Nix store on
their machine with all sources for a given nixpkgs checkout but also to ship
the sources of all successful hydra builds to some long-term storage.

It would be interesting to also measure/project the storage required to
archive all builds produced by hydra across time.

___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Set up a Sufficiently Powerful Build Farm

[Peter Simons]

Hi Eelco,

 >>> Just to be clear: you mean it compresses *to* 8%, or *by* 8%?
 >>
 >> By 8%.
 >
 > Hm, that's surprisingly low.

it turns out that nix-push compressed to ~8%, actually. I did not read
nix-push's out output correctly.

I repeated the experiment a moment ago, and a closure of all active
builds in 'haskellPackages' comes out weighing 2,212 MiByte (instead of
the uncompressed 31 GiByte) on x86_64-linux. I suppose this means the
build products from *all* Haskell package sets for *all* platforms would
take up approximately 100 GiByte in compressed form.

Best regards,
Peter

___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Set up a Sufficiently Powerful Build Farm

[Peter Simons]

Hi Eelco,

 > Regarding hydra.nixos.org disk space, I hope to address this by
 > having hydra-queue-runner store build results directly in an S3
 > bucket (i.e.cache.nixos.org), rather than in the local Nix store.

yes, that sounds like a good solution. It would be great to have that
feature available in hydra.nixos.org. Especially, if the change also
increases the performance of the evaluator.


 > Packages would appear in cache.nixos.org as soon as they have been
 > built, rather than when the channel mirror script runs.

Do you have any idea how to deal with the fact that some jobsets aren't
necessarily supposed to be available from cache.nixos.org, like
nixpkgs/multiple-outputs, because they're used for CI-testing work in
progress rather than for producing release binaries of Nixpkgs? There
seems to be an interesting problem insofar as that we don't know whether
a build run as part of "staging" is going to end up in the "unstable"
channel or not at the time when we run it. If the evaluation as a whole
is good, we merge "staging" to "master" and the build becomes a part of
Nixpkgs, but it might also happen that we have another stdenv change in
"staging" before such a merge occurs, and then the binaries produced by
that builds turn out to be irrelevant.


 > Do you have data on how well those packages compress? Cache.nixos.org
 > uses xz compression, so if hydra-queue-runner compressed build
 > results before uploading to S3, it would presumably need a lot less
 > than 1 TB.

I've nix-push'ed some 1,500 packages for testing purposes, and the tool
reports a total compression ratio of ~8%. That number is somewhat
inaccurate because nix-push publishes the *closure* of each store path I
give it -- so the compression ratio is computed over more than just the
Haskell packages, but still it feels like a reasonable result.

Generally speaking, the 1TB estimate is probably on the high end of the
scale, because it's based on the assumption that the size of the Haskell
package set for all platforms equals 3 times the size of that from
Linux/x86_64. In practice, however, i686 binaries will be smaller than
those from 86_64, and we also run far less builds on Darwin than on
Linux.

Anyway, I've also taken a moment to look at the build times of the
Haskell package set. On a moderately recent Linux/x86_64 server, the
build run-times (in minutes) are distributed as follows (based on
259,455 samples, not including "ghc" builds):

   Min.  1st Qu.   Median Mean  3rd Qu. Max.
0.09744  0.22790  0.37780  0.92970  0.75990 81.83000

The most expensive builds in terms of average run-time are:

  job builds   runtime medianmean sd
 1:  metadata 14 1145.6167 74.77 81.82976  23.462635
 2:poker-eval 35 2136.6000 48.50 61.04571  29.052927
 3:unicode-properties 13  458.9667 34.05 35.30513   8.226384
 4:   FpMLv53 19  652.8000 30.416667 34.35789  15.851984
 5:   texmath 91 3106.6000 21.55 34.13846  50.151076
 6:  scholdoc-texmath 77 2437.1500 32.00 31.65130   6.629542
 7:  amazonka-ec2109 3423.9500 28.63 31.41239  15.110833
 8: csound-expression-opcodes 27  846.7000  3.28 31.35926 142.592864
 9:  Agda 93 2860.5667 25.55 30.75878  14.754584
10: idris128 3269.7667 23.491667 25.54505   8.495109
11:accelerate-fourier 15  373.9167 31.18 24.92778  19.153399
12: uhc-light 46 1110.5333 22.95 24.14203   8.405919
13:   RSA 81 1940.1333  1.60 23.95226  56.325163
14:java-character 16  362.5500 23.70 22.65938   3.750131
15:  encoding 33  689.6333 17.83 20.89798   9.091368
16:  lens117 2421.4667 18.70 20.69630   8.762991
17: unicode-names 12  238.3833 20.058333 19.86528   1.812477
18:  hssqlppp 55 1019.6667 15.57 18.53939   6.277109
19:   uhc 21  369.0833 17.10 17.57540   5.794108
20:gl 74 1287.9167 17.875000 17.40428   5.709247
21:open-symbology 19  322.2667 16.98 16.96140   2.655905
22:  graphviz 53  894.6333 13.316667 16.87987  16.741832
23:  riak 47  782.3667  0.80 16.64610 108.513826
24:  haskell-src-exts 51  826.7000 14.53 16.20980   7.695385
25:   gtk 89 1408.8167 16.316667 15.82940   4.134229
26: git-annex510 7850.3833 13.891667 15.39291   6.257803
27:yi138 2101.7833 15.07 15.23031   3.314749
28:GenussFold 12  179.2333 14.941667 14.93611   4.535822
29:  midi 39  579.5667  2.97 14.86068  71.999306
30:  gtk3 89 1260.9667 

Re: [Nix-dev] Set up a Sufficiently Powerful Build Farm

[Eelco Dolstra]

Hi Peter,

On 04/11/15 14:52, Peter Simons wrote:

>  > Packages would appear in cache.nixos.org as soon as they have been
>  > built, rather than when the channel mirror script runs.
> 
> Do you have any idea how to deal with the fact that some jobsets aren't
> necessarily supposed to be available from cache.nixos.org, like
> nixpkgs/multiple-outputs, because they're used for CI-testing work in
> progress rather than for producing release binaries of Nixpkgs?

Everything would be stored in the binary cache right away, even packages that
don't end up in a NixOS/Nixpkgs channel. The main consequence is that we'd have
to run the binary cache garbage collection script [1] more frequently (currently
I manually run it once or twice a year).

[1]
https://github.com/NixOS/nixos-channel-scripts/blob/master/find-binary-cache-garbage.pl

>  > Do you have data on how well those packages compress? Cache.nixos.org
>  > uses xz compression, so if hydra-queue-runner compressed build
>  > results before uploading to S3, it would presumably need a lot less
>  > than 1 TB.
> 
> I've nix-push'ed some 1,500 packages for testing purposes, and the tool
> reports a total compression ratio of ~8%. 

Thanks!

Just to be clear: you mean it compresses *to* 8%, or *by* 8%?

-- 
Eelco Dolstra | LogicBlox, Inc. | http://nixos.org/~eelco/
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Set up a Sufficiently Powerful Build Farm

[Eelco Dolstra]

Hi,

On 04/11/15 17:28, Peter Simons wrote:
> Eelco Dolstra writes:
> 
>  > Just to be clear: you mean it compresses *to* 8%, or *by* 8%?
> 
> By 8%. "To" would be nice, though, wouldn't it? :-)

Hm, that's surprisingly low. (OTOH, compressing *to* 8% would be surprisingly
high...) For comparison, the store paths in my system closure obtained from the
binary cache compress to 25.3% (from 2882 MiB to 729 MiB).

-- 
Eelco Dolstra | LogicBlox, Inc. | http://nixos.org/~eelco/
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Set up a Sufficiently Powerful Build Farm

[Eelco Dolstra]

Hi,

On 29/10/15 13:38, Peter Simons wrote:

> Multiplying the average store path size by the number of distinct store paths
> tells us that storing *everything* requires approx. 360 GByte per platform.
> With 3 active platforms, we'd need about 1 TByte of disk space for one 
> complete
> set of Haskell packages.

Do you have data on how well those packages compress? Cache.nixos.org uses xz
compression, so if hydra-queue-runner compressed build results before uploading
to S3, it would presumably need a lot less than 1 TB.

-- 
Eelco Dolstra | LogicBlox, Inc. | http://nixos.org/~eelco/
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Set up a Sufficiently Powerful Build Farm

[Eelco Dolstra]

Hi Peter,

On 29/10/15 13:38, Peter Simons wrote:



Regarding hydra.nixos.org disk space, I hope to address this by having
hydra-queue-runner store build results directly in an S3 bucket (i.e.
cache.nixos.org), rather than in the local Nix store. (That would be part 2 of
the queue runner overhaul - part 1 was having a single-process queue runner that
loads the dependency graphs of all builds in the queue and can thus schedule
builds more efficiently.)

This would have a few advantages:

* Unlimited disk space.

* Remove the Nix store on the central machine as a bottleneck. Currently copying
store paths to/from the build slaves takes significantly more time than the
actual builds.

* Shorter latency - packages would appear in cache.nixos.org as soon as they
have been built, rather than when the channel mirror script runs.

The Nix store on the central machine would then only be used to communicate .drv
files between hydra-evaluator and hydra-queue-runner. But this shouldn't be much
data, so it would enable using a fast SSD drive for the Nix store.

Another recent improvement is hydra-provisioner, which is a script that fires up
EC2 spot instances dynamically based on the Hydra queue. This allows us to burn
through Linux builds pretty quickly. Thus the main problem right now is Darwin
builds, which are done on a single Mac mini...

> It's unclear whether the Hydra software would cope with 66 package sets with
> some 111,000 derivations in them that need to be evaluated, say, once an hour.
> Hydra has undergone some architectural changes recently that might make such a
> load possible -- i.e. "hydra-evaluator" is more efficient than it used to be
> --, but I don't have any reliable data concerning the performance of the
> process, so I cannot say what is possible and what is not.

Currently the main cause of evaluator slowness on hydra.nixos.org is due to the
high I/O load and the disk not being very fast in general. If the store, Nix
database and PostgreSQL database were on an SSD drive, I expect the evaluator to
be much faster.

-- 
Eelco Dolstra | LogicBlox, Inc. | http://nixos.org/~eelco/
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev