On Wed, 1 Jul 2020 at 08:09, Jan Kratochvil <jan.kratoch...@redhat.com> wrote:
> On Wed, 01 Jul 2020 13:43:57 +0200, Stephen John Smoogen wrote:
> > In the end we have limited resources and you are running into those
> > limits. We can either have fewer builders with more disk space and a
> > long queue of builds or we can have more builders with smaller disk
> > space.
> Good to know disk space really is a limiting factor there (as it is in general
> cheap compared to CPUs+RAM).

Disk space is only cheap if you don't care about the speed or the
amount of usable disk space you have. [You can have a system with 200
TB of disk space from a bunch of 4TB sata disks but find that only 20
TB is speedy and the rest has a fall over in performance because none
of the drives or controllers have enough cache] Every time we have
tried to get larger disks, the amount of complaints for slow builders
goes up exponentially. Trying for other disk layouts to increase IO
ends up with other downtime complaints (oh look RAID-10 failed on 2
drives in the same pair.. there goes everything).

Disk space which meets the IO needs of builders ends up being as
expensive as CPU cores and RAM amounts. The faster the IO needs, the
smaller the disk space per disk. You can fix this by having more
disks, but the more disks per system, the higher the power and cooling
costs per system.

In the end, there is no such thing as a free lunch.

> Sure a best step would be to have multiple (two) kinds of builders with
> different disk sizes and blacklist/whitelist packages to them. I am aware that
> idea is obvious, is there a Koji patch needed or even administration of two
> kinds of builders is too much?

I believe we have done this in the past for the x86_64 arch. The s390,
ppc64le, and aarch64 architectures have exceedingly limited resources
to try this with. At the moment, we are still trying to bring up what
we had in PHX2 to IAD2.

At the moment, the sysadmin team for Fedora are running on fumes. We
have worked several months of 10+ hour days+weekends to make the move
a success and we will be working at least a month more to bring up the
rest of the servers. At the moment we have about 10% of the hardware
we shipped back in operation and there are at least 2 to 3 weeks to
get the rest up. Getting up larger builders is going to have to wait.

Stephen J Smoogen.
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 

Reply via email to