On Thu, Oct 1, 2020 at 5:09 PM Shlomi Fish <shlo...@shlomifish.org> wrote: > On Tue, 29 Sep 2020 01:18:31 +0200 > Ole Tange <o...@tange.dk> wrote: > > > On Thu, Sep 24, 2020 at 8:37 AM Shlomi Fish <shlo...@shlomifish.org> wrote: : > > Unfortunately some of them are heavily dependent on running on the > > same equipment as mine due to timing issues (and every time I change > > hardware or upgrade software, I have to tweak the tests or results a > > little). > > I see. I suppose you are aware of running on a pristine CI system, e.g:
I had a talk with a guy who used Jenkins. Jenkins is all good and fine for tests that have predictable outcomes no matter the environment. It is worse when you have 2 tests: One has to run while the machine is swapping and one has to run when the machine is _not_ swapping. In other words they cannot be run in parallel. Or: A test has to run on a machine with high CPU load, while another test has to run on an idle machine. Or: A test has to be run on Centos 3, because other distributions do not show the error. I guess around 20% of my tests are in that category. The rest could probably be run by a normal CI. The current home-made test system also has the advantage that if a test fails, it will be pushed to the top, so when I run the test suite again, it will be the first test to run. > > If you want to help, a good start would be to have > > testsuite/REQUIREMENTS ported so it would work on more systems. It > > (tries to) install and set up all required software to run the > > testsuite. > > Wow, it looks intimidating. And I thought some of my FOSS projects had too > many > prereqs! The problem is that errors are often seen in the interaction with other tools - sometimes only as race conditions. /Ole
