Hi Ole!
Sorry for the late reply.
On Tue, 13 Oct 2020 13:56:39 +0200
Ole Tange <o...@tange.dk> wrote:
> On Thu, Oct 1, 2020 at 5:09 PM Shlomi Fish <shlo...@shlomifish.org> wrote:
> > On Tue, 29 Sep 2020 01:18:31 +0200
> > Ole Tange <o...@tange.dk> wrote:
> >
> > > On Thu, Sep 24, 2020 at 8:37 AM Shlomi Fish <shlo...@shlomifish.org>
> > > wrote:
> :
> > > Unfortunately some of them are heavily dependent on running on the
> > > same equipment as mine due to timing issues (and every time I change
> > > hardware or upgrade software, I have to tweak the tests or results a
> > > little).
> >
> > I see. I suppose you are aware of running on a pristine CI system, e.g:
>
> I had a talk with a guy who used Jenkins. Jenkins is all good and fine
> for tests that have predictable outcomes no matter the environment.
>
> It is worse when you have 2 tests: One has to run while the machine is
> swapping and one has to run when the machine is _not_ swapping. In
> other words they cannot be run in parallel.
>
> Or: A test has to run on a machine with high CPU load, while another
> test has to run on an idle machine.
>
> Or: A test has to be run on Centos 3, because other distributions do
> not show the error.
>
> I guess around 20% of my tests are in that category. The rest could
> probably be run by a normal CI.
I see. I tend to agree.
>
> The current home-made test system also has the advantage that if a
> test fails, it will be pushed to the top, so when I run the test suite
> again, it will be the first test to run.
>
This is possible to do in CI setups as well, with a (hopefully) small amount of
scripting.
> > > If you want to help, a good start would be to have
> > > testsuite/REQUIREMENTS ported so it would work on more systems. It
> > > (tries to) install and set up all required software to run the
> > > testsuite.
> >
> > Wow, it looks intimidating. And I thought some of my FOSS projects had too
> > many prereqs!
>
> The problem is that errors are often seen in the interaction with
> other tools - sometimes only as race conditions.
>
I see - we can try modularising it.
Regards,
Shlomi
>
> /Ole
>
--
Shlomi Fish https://www.shlomifish.org/
Summer Glau Facts - https://shlom.in/sglau-facts
Hacker sees bug. Hacker does not want bug. Hacker fixes bug.
— https://www.shlomifish.org/humour.html
Please reply to list if it's a mailing list post - https://shlom.in/reply .
