Hi Tim, On Fri, Sep 06, 2019 at 04:30:24PM +0200, Tim Düsterhus wrote: > Dear List > > something between 02bac85bee664976f6dcecc424864e9fb99975be and > f909c91e8a739b9ef7409b399259201fe883771c broke all the Linux builds on > Travis CI: > > - 41 reg tests fail with a timeout > - 3 reg tests pass > > FreeBSD works fine. > > Somebody really ought to take a look. I might try to bisect if I find a > bit of spare time. If someone beats me to it: Go ahead.
I've been quite annoyed with this a number of times and ended up not looking at build reports anymore due to this. I've spent some time looking at the cause as well and bisecting, coming to the conclusion that apparently the travis VMs are regularly overloaded. Most of the times we see TCP connection timeouts on the loop back preventing the vtest client from reaching haproxy! I've even seen a number of "out of memory" messages hitting the client. It's possible that their hypervisor is sometimes running out of memory. Maybe their service is abused by other projects which induce a huge load. At some point I used to click "build again", which managed to randomly work, but I gave up, being used to seeing this constantly red :-( I think that we should simply disable reg tests and stick to build tests only. There's nothing worse than getting used to seeing errors, as by not seeing a difference between a build error and a test error we get trained to ignore results. Maybe we can keep the reg tests for cron jobs, but given that they similarly fail I don't see the benefit either. I too would like to see them turn green again :-/ Cheers, Willy
