Hi,
On 8/4/20 2:29 AM, Kevin Fenzi wrote:
ok. I did what I could with the resources we have right now to improve
things on the s390x builders.
1. I noticed that we had the kvm instances oversubscribed on cpus (the
host has 32, we had 42 used). So, I lowered all the kvm builders to 3
vcpus from 4. (Those are 15-24).
2. I moved the varnish package cache from 07 (a z/vm guest) to 24 (a kvm
guest). I have noticed the z/vm ones (thats 01-14) seem to suffer from
slowdowns or high io wait more under high load and/or over a long time.
Hopefully moving that to a more stable instance will help with lots of
issues people have seen with not being able to download or the like.
3. I switched the cache model on the kvm ones to unsafe, which we had
already used on a number of other builders. I think the worst that can
happen here is that the vm becomes corrupt if it's abruptly shutdown or
killed, but thats fine, we can just spin up a new one. If a build gets
messed up, koji would just restart it again on another vm, etc.
4. There was a misconfiguration in kojid where if the cache was not
answering it tried directly, but it was trying the wrong url. I have
corrected this, so if the primary cache is down it should fall back to
trying directly on it's own.
5. I've updated and rebooted them all. They seem to behave much better
after reboots and then slowly get slower over time. ;(
I've been watching it for the last hour or so and so far 0 failures that
I can attribute to s390x cache or builder infra.
Hopefully that should make things more stable.
Awesome, thank you!
Regards,
Hans
_______________________________________________
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
