koji hubs and builders updated

2022-01-16 Thread Kevin Fenzi
Just thought I would share that (finally) we have completed upgrades on
the koji hubs and builders.

The hubs have been upgraded to Fedora 35 and
the latest koji version (1.17.1).

Builder hypervisors have been upgraded to RHEL8.5 and the latest
advanced virt stack. Builders have been upgraded/reimaged to Fedora 35
and the latest packages. 

On armv7 builders: As you all know due to 
https://bugzilla.redhat.com/show_bug.cgi?id=1920183
( Overeager OOM kills on 5.10 kernels on 32bit arm with lpae )
we had been running an old kernel on them, but after some testing in
staging it seemed that the upcoming 5.16 kernel worked pretty well. 
So, All of them are running that now. There have been some OOM kills
already, I'm playing around with options on them to try and get that
number to 0. Hopefully they will be ok for the mass rebuild. 

On s390x builders: We got some more resources, so the kvm builders now
have more memory and cpus. I've asked for some more disk space and once
we have that I plan to spread those resources out to have 2x as many
builders (ie, now: 4 cpus, 20gb mem x 10, after: 2 cpus, 10gb mem x 20)
I don't know if this will be done in time for the mass rebuild, but
hopefully so. 

On ppc64le: We have some more power9 hardware waiting to be put in
service, so we should be adding some builders here, and/or spreading out
resources. 

Thanks for everyone's patience with builds that have been taking a while
or restarting. Do continue to report them and we will do the best we can
to fix things. 

kevin


signature.asc
Description: PGP signature
___
devel-announce mailing list -- devel-annou...@lists.fedoraproject.org
To unsubscribe send an email to devel-announce-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel-annou...@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure


koji hubs and builders updated

2022-01-16 Thread Kevin Fenzi
Just thought I would share that (finally) we have completed upgrades on
the koji hubs and builders.

The hubs have been upgraded to Fedora 35 and
the latest koji version (1.17.1).

Builder hypervisors have been upgraded to RHEL8.5 and the latest
advanced virt stack. Builders have been upgraded/reimaged to Fedora 35
and the latest packages. 

On armv7 builders: As you all know due to 
https://bugzilla.redhat.com/show_bug.cgi?id=1920183
( Overeager OOM kills on 5.10 kernels on 32bit arm with lpae )
we had been running an old kernel on them, but after some testing in
staging it seemed that the upcoming 5.16 kernel worked pretty well. 
So, All of them are running that now. There have been some OOM kills
already, I'm playing around with options on them to try and get that
number to 0. Hopefully they will be ok for the mass rebuild. 

On s390x builders: We got some more resources, so the kvm builders now
have more memory and cpus. I've asked for some more disk space and once
we have that I plan to spread those resources out to have 2x as many
builders (ie, now: 4 cpus, 20gb mem x 10, after: 2 cpus, 10gb mem x 20)
I don't know if this will be done in time for the mass rebuild, but
hopefully so. 

On ppc64le: We have some more power9 hardware waiting to be put in
service, so we should be adding some builders here, and/or spreading out
resources. 

Thanks for everyone's patience with builds that have been taking a while
or restarting. Do continue to report them and we will do the best we can
to fix things. 

kevin


signature.asc
Description: PGP signature
___
devel-announce mailing list -- devel-announce@lists.fedoraproject.org
To unsubscribe send an email to devel-announce-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure