On 2/4/26 10:26 AM, Ben Cooksley wrote:
On Wed, Feb 4, 2026 at 9:07 PM Ben Cooksley <[email protected]> wrote:
On Wed, Feb 4, 2026 at 8:25 PM Vlad Zahorodnii
<[email protected]> wrote:
Hello,
HI Vlad,
Over approximately the past weekend something happened in our
CI and now
it takes quite long time for tests to run. For example, in
kwin, we have
a test that used to run for about 20 seconds, and now it takes
about 5
or so minutes to finish running. Speaking for kwin, there were no
changes that could increase test run times so dramatically.
January 26th:
Start 61: kwin-testOutputChanges
61/158 Test #61: kwin-testOutputChanges
............................. Passed 19.36 sec
January 29th:
Start 61: kwin-testOutputChanges
61/158 Test #61: kwin-testOutputChanges
............................. Passed 43.93 sec
January 30th:
Start 61: kwin-testOutputChanges
61/158 Test #61: kwin-testOutputChanges
............................. Passed 45.91 sec
Februrary 3rd:
Start 61: kwin-testOutputChanges
61/158 Test #61: kwin-testOutputChanges
............................. Passed 254.19 sec
FreeBSD appears to be fine.
We suspect that test run times blew up due to enabling LSAN in
various
libraries (kwin itself has no LSAN enabled yet). The issue
doesn't
appear to be specific to only kwin, people reported that
they've seen
similar issues in other projects too. Maybe something else
happened to
CI that sysadmins will be able to clarify.
Nothing else happened to CI recently aside from the enablement of
LSAN.
The underlying SUSE images were for Qt 6.10 at least last rebuilt
on January 25th, which is well before your "last good" date.
The only change to CI between January 30th and February 3rd
was fast_unwind_on_malloc=0 being added by default, even though it
is primarily for the benefit of LSAN.
I've now made changes to only set fast_unwind_on_malloc=0 if LSAN
is explicitly enabled for a repository - hard to tell if that will
fix the issue though as KWin takes a while to build.
For the record, as per
https://invent.kde.org/plasma/kwin/-/jobs/3958895 which completed
moments ago:
That branch contained some other things that could interfere with test
results. I started https://invent.kde.org/plasma/kwin/-/jobs/3959078 and
yeah it looks like test run times are back to the Jan 29-30th level.
So, it seems like fast_unwind_on_malloc=0 is the culprit then?
Regards,
Vlad
Start 61: kwin-testOutputChanges
61/158 Test #61: kwin-testOutputChanges
............................. Passed 19.58 sec
Either way, the current state of CI is not great.
Hypothetically, test
timeouts can be increased but QSignalSpy's have hardcoded
timeouts that
can be too low for the current CI. And in case of kwin, 5
minutes for a
test is simply not a workable thing.
Regards,
Vlad
Thanks,
Ben
Cheers,
Ben
