After I increased capacity it started to pick tasks, but all were stuck, I tracked it down to NFS hard connection which was stuck, I rebooted the builder and it does not come up.
If I won't fix this in a reasonable time, I will shutdown htttp on our new koji to prevent more tasks to queue up. NFS was always huge pain of our deployment and I think we can simply launch koji on new but bigger instance with enough memory and CPUs so we can handle all the load with single instance without NFS dance. LZ On Fri, Jun 9, 2017 at 10:17 AM, Lukas Zapletal <[email protected]> wrote: > Hey, > > we are suffering from capacity issues - builder3 (that's the only > extra builder we currently have) was spawned on hypervisor which is > under stress, therefore kernel reports load of 3 and kojid does not > schedule any builds there since this exceeds the capacity. The > instance is simply idling, I need to figure out how to solve this by > either increasing capacity or restarting it on different hypervisor. > > LZ > > On Sun, Jun 4, 2017 at 1:34 PM, Lukas Zapletal <[email protected]> wrote: >> Hey, >> >> I had to brought Koji services offline, I expect them to come back in >> an hour, we are trying to move the data to different EC2 instance. >> This is just a test and if the new instance will work fine, we will do >> the switchover during the week. >> >> Sorry for inconvenience, please resubmit your jobs if you have problems. >> >> -- >> Later, >> Lukas @lzap Zapletal > > > > -- > Later, > Lukas @lzap Zapletal -- Later, Lukas @lzap Zapletal -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
