Re: [racket-dev] CI improved for Racket

['Paulo Matos' via Racket Developers]

On 11/04/2019 11:56, Jack Firth wrote:
> So about 30-40 total cores for that second category?

That would be awesome! :)

> About how much
> total RAM is needed? 

Someone might correct me here but from what I can see it would be great
to have something like 2G/core - but if not it shouldn't be a problem.
One can always force certain jobs to certain hosts through the use of
tags so you can run 2 RAM hungry jobs or 1 RAM hungry job and 4 where
RAM doesn't matter so much.

> I'm assuming Gitlab owns all persisted data so
> there's not much need for these machines to have non-transient disk
> storage. 

No, currently I have a central cache for the gitlab machines locally.
However, I have in place an S3 bucket (Google would also be possible but
would need to check) I will sponsor to hold the cache between the machines.

> How much importance does resource density play, in terms of
> ideal cores-and-RAM-per-machine? Can you run the CI tasks effectively
> across several (>=8 nodes) small machines (<=4 cores, 4GB RAM) instead
> of a few big ones? How bursty is the CI workload, and would the machines
> spend a lot of time sitting idle?
> 

So here it depends. There are 14 jobs at the moment that cannot go to
the free gitlab machines. These are building Racket in one way or
another. I would say best is to build Racket with at least 4 cores, so
depending on the available hardware I would split the jobs the best
possible way. With a single 20 cores machines with 64G RAM I could ran 5
builds/jobs in parallel but however if I only had 16G RAM than it might
be better not to do 5 in parallel. If RAM is limited per machine, it's
better to have more machines. The machines will sit idle sometimes but
not often because some jobs just take a long time. So if Matthew pushes
something every two hours, jobs start to queue up. They might sit idle
during weekends though when people don't generally work.

If you have a specific suggestion then I could tell you what would work
and what wouldn't but in any case, I need to already say thank you for
your interest in helping the CI process.

> It might be feasible for me to donate a Kubernetes cluster to the cause
> (but I make no promises).
> 

Thanks. I don't know much about kubernetes (still at the docker level...
:)) but I saw the name thrown around in the gitlab docs so it should be
fine integrating such a cluster with CI.

Paulo

> On Thu, Apr 11, 2019 at 2:21 AM Paulo Matos <pmatos@linki.tools> wrote:
> 
> 
> 
>     On 11/04/2019 11:01, jackhfi...@gmail.com
>     <mailto:jackhfi...@gmail.com> wrote:
>     > On Wednesday, April 10, 2019 at 12:15:02 AM UTC-7, Paulo Matos wrote:
>     >
>     >     Currently I don't have enough machines or AWS time to dedicate to
>     >     Racket builds
>     >
>     >
>     > How much do you need?
>     >
> 
>     There are really two types of needs here.
>      - Different architecture machines: ppc64, mips32, mips64, arm
>      - x86_64 with several cores - speaking in terms of cores, having 16
>     more cores would be great. Also the machines wouldn't need to be on all
>     the time if not possible. Lets say you have a spare machines, you could
>     start the gitlab-runner only during certain periods and gitlab would
>     schedule the jobs appropriately.
> 
>     With regards to the number of cores, the more, the merrier. Running qemu
>     to emulate the architectures above is slow and a lot of the jobs have to
>     wait because I only have 2 machines (totalling 20 cores) assigned to
>     this and there are lots of jobs that need to run.
> 
>     Regards,
> 
>     -- 
>     Paulo Matos
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "Racket Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to racket-dev+unsubscr...@googlegroups.com
> <mailto:racket-dev+unsubscr...@googlegroups.com>.
> To post to this group, send email to racket-dev@googlegroups.com
> <mailto:racket-dev@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/racket-dev/CAAXAoJWJJJb39Ec1MN78MzKL0SDgJ7TEAj9QkV%3DXVRxodfP6Uw%40mail.gmail.com
> <https://groups.google.com/d/msgid/racket-dev/CAAXAoJWJJJb39Ec1MN78MzKL0SDgJ7TEAj9QkV%3DXVRxodfP6Uw%40mail.gmail.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

-- 
Paulo Matos

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-dev+unsubscr...@googlegroups.com.
To post to this group, send email to racket-dev@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/racket-dev/7a136b88-a9e8-f30f-9f04-6306efc489be%40linki.tools.
For more options, visit https://groups.google.com/d/optout.