On Wed, Feb 4, 2026 at 10:05 PM Vlad Zahorodnii <[email protected]> wrote:
> On 2/4/26 10:26 AM, Ben Cooksley wrote: > > On Wed, Feb 4, 2026 at 9:07 PM Ben Cooksley <[email protected]> wrote: > >> On Wed, Feb 4, 2026 at 8:25 PM Vlad Zahorodnii <[email protected]> >> wrote: >> >>> Hello, >>> >> >> HI Vlad, >> >> >>> >>> Over approximately the past weekend something happened in our CI and now >>> it takes quite long time for tests to run. For example, in kwin, we have >>> a test that used to run for about 20 seconds, and now it takes about 5 >>> or so minutes to finish running. Speaking for kwin, there were no >>> changes that could increase test run times so dramatically. >>> >>> January 26th: >>> >>> Start 61: kwin-testOutputChanges >>> 61/158 Test #61: kwin-testOutputChanges >>> ............................. Passed 19.36 sec >>> >>> January 29th: >>> >>> Start 61: kwin-testOutputChanges >>> 61/158 Test #61: kwin-testOutputChanges >>> ............................. Passed 43.93 sec >>> >>> January 30th: >>> >>> Start 61: kwin-testOutputChanges >>> 61/158 Test #61: kwin-testOutputChanges >>> ............................. Passed 45.91 sec >>> >>> Februrary 3rd: >>> >>> Start 61: kwin-testOutputChanges >>> 61/158 Test #61: kwin-testOutputChanges >>> ............................. Passed 254.19 sec >>> >>> FreeBSD appears to be fine. >>> >>> We suspect that test run times blew up due to enabling LSAN in various >>> libraries (kwin itself has no LSAN enabled yet). The issue doesn't >>> appear to be specific to only kwin, people reported that they've seen >>> similar issues in other projects too. Maybe something else happened to >>> CI that sysadmins will be able to clarify. >>> >> >> Nothing else happened to CI recently aside from the enablement of LSAN. >> >> The underlying SUSE images were for Qt 6.10 at least last rebuilt on >> January 25th, which is well before your "last good" date. >> >> The only change to CI between January 30th and February 3rd >> was fast_unwind_on_malloc=0 being added by default, even though it is >> primarily for the benefit of LSAN. >> I've now made changes to only set fast_unwind_on_malloc=0 if LSAN is >> explicitly enabled for a repository - hard to tell if that will fix the >> issue though as KWin takes a while to build. >> > > For the record, as per https://invent.kde.org/plasma/kwin/-/jobs/3958895 > which completed moments ago: > > That branch contained some other things that could interfere with test > results. I started https://invent.kde.org/plasma/kwin/-/jobs/3959078 and > yeah it looks like test run times are back to the Jan 29-30th level. > > So, it seems like fast_unwind_on_malloc=0 is the culprit then? > Yes, quoting from https://source.android.com/docs/security/test/asan [quote] ASan uses a fast, frame-pointer-based unwinder to record a stack trace for every memory allocation and deallocation event in the program. Most of Android is built without frame pointers. As a result, you often get only one or two meaningful frames. To fix this, either rebuild the library with ASan (recommended!), or with: LOCAL_CFLAGS:=-fno-omit-frame-pointer LOCAL_ARM_MODE:=arm Or set ASAN_OPTIONS=fast_unwind_on_malloc=0 in the process environment. The latter can be very CPU-intensive, depending on the load. [/quote] That would align with what we're seeing here Regards, > Vlad > Cheers, Ben > > Start 61: kwin-testOutputChanges > 61/158 Test #61: kwin-testOutputChanges ............................. > Passed 19.58 sec > > >> >> >>> >>> Either way, the current state of CI is not great. Hypothetically, test >>> timeouts can be increased but QSignalSpy's have hardcoded timeouts that >>> can be too low for the current CI. And in case of kwin, 5 minutes for a >>> test is simply not a workable thing. >>> >>> Regards, >>> Vlad >>> >> >> Thanks, >> Ben >> > > Cheers, > Ben > >
