Re: Fixing tests failing on CI but not on local system

[Amber Schenck]

On 4/23/25 2:53 AM, Stefano Crocco wrote:

On martedì 22 aprile 2025 20:57:53 Ora legale dell’Europa centrale Ben
Cooksley wrote:

On Wed, Apr 23, 2025 at 5:24 AM Stefano Crocco <stefano.cro...@alice.it>

wrote:

Hello to everyone,

Hi Stefano,

Hello Ben,

recently, I've being trying to fix failing Konqueror tests, and in doing
so I'm
facing some issue with the CI tests which I don't know how to solve.

The first issue is that one test passes on my system but fails when run by
the
CI (it's the konqview test at [1]). I couldn't spot anything in the code
which
would be likely to be different on the CI than on my system. What do you
think
is the best way to try and fix this test? Should I try and add some
debugging
statement in the tests, then run it again on the CI to try to pinpoint the
cause of the failure? I wouldn't want to waste CI resources, but I have no
better idea.

This test looks like it might be a KXMLGUI initialisation failure:

QWARN  : KonqViewTest::textThenHtml() kf.xmlgui: cannot find .rc file
"konqueror.rc" for component "konqviewtest"
QWARN  : KonqViewTest::textThenHtml() kf.xmlgui: cannot find .rc file
"konqueror.rc" for component "konqviewtest"

It is probably masked on your local system because there will be old files
/ configuration / etc present.
Might be worth creating a new user account, doing a build of Konqueror
alone there (without logging in or running Konqueror normally there) and
seeing if the unit test fails there as that is basically what CI does.

I don't think this can be the difference because I get the same message on my
system where the test succeeds (because the konqueror.rc file is installed in
the konqueror directory, while the KXmlGui frameworks expects it in the
konqviewtest directory since that's the name of the component) . Just to be
sure, I tried following your suggestion but nothing changed and the test
succeeded.

Happy Testing and Good Luck!
-Amber

We were writing tests for code that would write to /etc in production and of course didn't want that to happen in the test case, so we used the sandbox tool from gentoo[1] to ensure that it would only read/write from the test data dir instead of /etc. It also conveniently showed that it was reading from ~/.config/ which could have led to inconsistent results on different systems. That at least will show a stack trace where in the code the read is coming from, which is a hint to start tracking how to prevent it from happening in tests. For our tests the solution was to use QTEST_APPLESS_MAIN instead of QTEST_MAIN. 1: https://wiki.gentoo.org/wiki/Project:Sandbox>>> A similar issue regards tests failing on FreeBSD but not on Linux: how

should
I proceed, since that I have no FreeBSD machine available and I'm not
familiar
enough with it to try setting up a virtual machine?

In the not too distant future once we have VM based CI you'd be able to
download the VM image the CI system uses.
This is weeks, not months, away at this point.

This should be very useful. Thanks for the information.

Lastly, there's a test (konqviewmgrtest at [2]) which is marked as failed
by
the CI but, looking at the output it reports, seems successful to me (I
can't
see any mention of failing tests or errors in the output). I attach a file
with
the output of the test. My only guess is that the word "failed" which
appears
in lines 3 and 28 but which is unrelated to the test itself may be
confusing
the CI. Do you think it is possible? Otherwise, do you have any idea about
why
the test could be marked as failed?

Reading the full build log reveals why that happened:

4/12 Test #4: konqviewmgrtest ...................***Timeout 60.12 sec

My guess would be that it is popping up a message box as something didn't
quite work out when it tries to activate the link:

QDEBUG : ViewMgrTest::testLinkedViews() ACTIVATING LINK

Thanks for pointing this out. I didn't think of looking at the full output.
I'll try to find out could be producing a message box, but it won't be easy.

Thanks in advance

Stefano

Hope that gives you some ideas!

Cheers,
Ben

Thanks

Stefano

--
Attached is my PGP public key.
Primary key fingerprint: 4407 5FB3 3665 0970 3B75 CD31 7DA1 7F4D AC46 7943

If you have a PGP key (and a minute to spare)
please send it in reply to this email.

If you have no idea what PGP is, feel free
to ignore all this gobbledegook.

OpenPGP_0x7DA17F4DAC467943.asc
Description: OpenPGP public key

OpenPGP_signature.asc
Description: OpenPGP digital signature