On 2/4/26 10:16 AM, Ben Cooksley wrote:
On Wed, Feb 4, 2026 at 10:05 PM Vlad Zahorodnii
<[email protected]> wrote:
On 2/4/26 10:26 AM, Ben Cooksley wrote:
On Wed, Feb 4, 2026 at 9:07 PM Ben Cooksley <[email protected]>
wrote:
On Wed, Feb 4, 2026 at 8:25 PM Vlad Zahorodnii
<[email protected]> wrote:
Hello,
HI Vlad,
Over approximately the past weekend something happened in
our CI and now
it takes quite long time for tests to run. For example,
in kwin, we have
a test that used to run for about 20 seconds, and now it
takes about 5
or so minutes to finish running. Speaking for kwin, there
were no
changes that could increase test run times so dramatically.
January 26th:
Start 61: kwin-testOutputChanges
61/158 Test #61: kwin-testOutputChanges
............................. Passed 19.36 sec
January 29th:
Start 61: kwin-testOutputChanges
61/158 Test #61: kwin-testOutputChanges
............................. Passed 43.93 sec
January 30th:
Start 61: kwin-testOutputChanges
61/158 Test #61: kwin-testOutputChanges
............................. Passed 45.91 sec
Februrary 3rd:
Start 61: kwin-testOutputChanges
61/158 Test #61: kwin-testOutputChanges
............................. Passed 254.19 sec
FreeBSD appears to be fine.
We suspect that test run times blew up due to enabling
LSAN in various
libraries (kwin itself has no LSAN enabled yet). The
issue doesn't
appear to be specific to only kwin, people reported that
they've seen
similar issues in other projects too. Maybe something
else happened to
CI that sysadmins will be able to clarify.
Nothing else happened to CI recently aside from the
enablement of LSAN.
The underlying SUSE images were for Qt 6.10 at least last
rebuilt on January 25th, which is well before your "last
good" date.
The only change to CI between January 30th and February 3rd
was fast_unwind_on_malloc=0 being added by default, even
though it is primarily for the benefit of LSAN.
I've now made changes to only set fast_unwind_on_malloc=0 if
LSAN is explicitly enabled for a repository - hard to tell if
that will fix the issue though as KWin takes a while to build.
For the record, as per
https://invent.kde.org/plasma/kwin/-/jobs/3958895 which completed
moments ago:
That branch contained some other things that could interfere with
test results. I started
https://invent.kde.org/plasma/kwin/-/jobs/3959078 and yeah it
looks like test run times are back to the Jan 29-30th level.
So, it seems like fast_unwind_on_malloc=0 is the culprit then?
Yes, quoting from https://source.android.com/docs/security/test/asan
[quote]
ASan uses a fast, frame-pointer-based unwinder to record a stack trace
for every memory allocation and deallocation event in the program.
Most of Android is built without frame pointers. As a result, you
often get only one or two meaningful frames. To fix this, either
rebuild the library with ASan (recommended!), or with:
LOCAL_CFLAGS:=-fno-omit-frame-pointer
LOCAL_ARM_MODE:=arm
Or set ASAN_OPTIONS=fast_unwind_on_malloc=0 in the process
environment. The latter can be very CPU-intensive, depending on the load.
[/quote]
That would align with what we're seeing here
Hi,
thanks for investigating this.
I'm a bit surprised that fast_unwind_on_malloc seems to have an effect
when LSAN is disabled. That said, we still should have the problem when
LSAN *is* enabled, no?
The reason I added fast_unwind_on_malloc=0 is because otherwise we would
sometimes get unusable backtraces from LSAN, so I would not want to lose
that.
A possible remedy could be to build Qt with ASAN support. That's one
additional flag in the buildsystem invokation. fast_unwind_on_malloc
still would make sense to cover system libraries, but maybe it would be
okay to do without that.
Cheers
Nico
