There is a bad news and a good news :). * The bad one is that the change did not go well with its original scope. It turned out that many small jobs are not a good idea when you have 180 slots in a queue and a number (growing) of Apache projects and yours are competing for those. Seems that our jobs got starved a lot and the effect was 2-3 hours waiting queues which were growing afternoon when US started to wake up.
* The good one is that I just merged a fix to that - instead of many small jobs, we grouped several test types in single jobs and we clean-up between the jobs and reusing the machines. I believe this will be even more optimized, and uses the same concepts of optimization as before. I cancelled all the queued builds and asked people to rebase to latest master. If you have not done so yet - please do it now! J. On Mon, Oct 12, 2020 at 1:27 AM Daniel Imberman <[email protected]> wrote: > Thanks Jarek! This was much needed and should lead to a cleaner dev process > > via Newton Mail > <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.51&pv=10.15.6&source=email_footer_2> > > On Sun, Oct 11, 2020 at 3:37 PM, Jarek Potiuk <[email protected]> > wrote: > > Hello everyone, > > I have really high hopes for the CI change that we implemented over the > weekend. Last few weeks we experienced a lot of stability problems with the > CI, and our builds were rarely "green" - and mostly due to > intermittent/unrelated problems. We've implemented some workarounds and > splitting to a bigger number of smaller jobs that so far has proven to be > much more stable and "greener", > > You will see a much bigger number of test checks than you used to (up to > 120 or so), but they will be quite a bit faster. Also - if any of the > checks fail fo a good reason, you should be able to find information on how > to reproduce the failures locally in the test output - so that you can fix > it. > > We will be watching and fixing any teething problems over the next few > days, but for now - please rebase to the latest master and try it out. > > J. > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
