[Desktop-packages] [Bug 1785262] Re: arm64 autopkgtests are flaky
Groovy verified per LP: #1913353. ** Tags removed: verification-needed-groovy ** Tags added: verification-done-groovy -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to libreoffice in Ubuntu. https://bugs.launchpad.net/bugs/1785262 Title: arm64 autopkgtests are flaky Status in LibreOffice: Confirmed Status in libreoffice package in Ubuntu: Fix Released Status in libreoffice source package in Groovy: Fix Released Bug description: [Verification and regression analysis is all handled via the main SRU bug of LP: #1913353] See http://autopkgtest.ubuntu.com/packages/libreoffice/cosmic/arm64. In the past 30 days, 8 out of 45 test runs failed. That a 18% failure rate. It would be good to understand if all those failures are similar, what's causing them, and if they can easily be made more robust. To manage notifications about this bug go to: https://bugs.launchpad.net/df-libreoffice/+bug/1785262/+subscriptions -- Mailing list: https://launchpad.net/~desktop-packages Post to : desktop-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~desktop-packages More help : https://help.launchpad.net/ListHelp
[Desktop-packages] [Bug 1785262] Re: arm64 autopkgtests are flaky
This bug was fixed in the package libreoffice - 1:7.0.5-0ubuntu0.20.10.1 --- libreoffice (1:7.0.5-0ubuntu0.20.10.1) groovy; urgency=medium * New upstream release (LP: #1913353) [ Rico Tzschichholz ] * Update yaru icon style "2021-01-31" * Move libreoffice-sdbc-hsqldb to Recommends (LP: #1916786) * Fix Calc crash in ScSelectionTransferObj (LP: #1897784) * Drop fix-bluez-external.diff and unowinreg-static-libgcc.diff, applied upstream [ Heather Ellsworth ] * Add timeout to uicheck-sw test to allow it to handle timeouts under load better (LP: #1785262) libreoffice (1:7.0.4-3) unstable; urgency=medium * debian/tests/control.in: *really* add libreoffice-writer dependency to uicheck-sc test libreoffice (1:7.0.4-2) unstable; urgency=medium * debian/test/control: make uicheck-sc depend on libreoffice-writer, too (the openDialogs/uno.Show:License Dialog test opens a new "Writer/Web" document...) libreoffice (1:7.0.4-1) unstable; urgency=medium * LibreOffice 7.0.4 final release (identical to rc2) * debian/patches/pdfium-m68k.diff: fix pdfium build on m68k * debian/rules, debian/control*in: s/noinsttests/noinsttest/, thanks lintian * debian/rules: - revert clang (>= 1:11) build-dep for buster-backports; doesn't exist in buster and we resort back to gcc - don't rm LICENSE.html, it is used by Help -> License Information -> Show License * debian/control.mediawiki.in: remove Homepage: (closes: #978713) * debian/*.mime: stop quoting %s (closes: #950319) libreoffice (1:7.0.4~rc2-1) unstable; urgency=medium * New upstream release candidate libreoffice (1:7.0.4~rc1-1) unstable; urgency=medium * New upstream release candidate * debian/patches/unowinreg-static-libgcc.diff: use -static-libgcc, see LO master commit 01241113947fc7bd7f7b765dd897bb023c8ca99c * debian/rules: - set MYSQL_FLAVOUR=mariadb (as it's used anyway, and upstreams internal copy is mariadb-connector-c anyway) and build-depend on libmariadb-dev instead of libmariadbclient-dev-compat (closes: #975481) - build-depend on clang (>= 1:11) on armel - fix nocheck -b builds... - use nowindows build profile for (not) building unowinreg * debian/rules, debian/control*in: - honour noinsttests build profile for (not) building -subsequentcheckbase/-smoketest-test-data * debian/rules, debian/control*in: - move unowinreg.dll into /usr/lib/i686-w64-mingw32 (the mingw g++ compiler installs its .dlls into /usr/lib/gcc/i686-w64-mingw32) -- Rico Tzschichholz Mon, 15 Mar 2021 19:01:12 +0100 ** Changed in: libreoffice (Ubuntu Groovy) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to libreoffice in Ubuntu. https://bugs.launchpad.net/bugs/1785262 Title: arm64 autopkgtests are flaky Status in LibreOffice: Confirmed Status in libreoffice package in Ubuntu: Fix Released Status in libreoffice source package in Groovy: Fix Released Bug description: [Verification and regression analysis is all handled via the main SRU bug of LP: #1913353] See http://autopkgtest.ubuntu.com/packages/libreoffice/cosmic/arm64. In the past 30 days, 8 out of 45 test runs failed. That a 18% failure rate. It would be good to understand if all those failures are similar, what's causing them, and if they can easily be made more robust. To manage notifications about this bug go to: https://bugs.launchpad.net/df-libreoffice/+bug/1785262/+subscriptions -- Mailing list: https://launchpad.net/~desktop-packages Post to : desktop-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~desktop-packages More help : https://help.launchpad.net/ListHelp
[Desktop-packages] [Bug 1785262] Re: arm64 autopkgtests are flaky
Hello Olivier, or anyone else affected, Accepted libreoffice into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libreoffice/1:7.0.5-0ubuntu0.20.10.1 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed- groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification- failed-groovy. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: libreoffice (Ubuntu Groovy) Status: New => Fix Committed ** Tags added: verification-needed verification-needed-groovy -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to libreoffice in Ubuntu. https://bugs.launchpad.net/bugs/1785262 Title: arm64 autopkgtests are flaky Status in LibreOffice: Confirmed Status in libreoffice package in Ubuntu: Fix Released Status in libreoffice source package in Groovy: Fix Committed Bug description: [Verification and regression analysis is all handled via the main SRU bug of LP: #1913353] See http://autopkgtest.ubuntu.com/packages/libreoffice/cosmic/arm64. In the past 30 days, 8 out of 45 test runs failed. That a 18% failure rate. It would be good to understand if all those failures are similar, what's causing them, and if they can easily be made more robust. To manage notifications about this bug go to: https://bugs.launchpad.net/df-libreoffice/+bug/1785262/+subscriptions -- Mailing list: https://launchpad.net/~desktop-packages Post to : desktop-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~desktop-packages More help : https://help.launchpad.net/ListHelp
[Desktop-packages] [Bug 1785262] Re: arm64 autopkgtests are flaky
** Description changed: + [Verification and regression analysis is all handled via the main SRU + bug of LP: #1913353] + See http://autopkgtest.ubuntu.com/packages/libreoffice/cosmic/arm64. In the past 30 days, 8 out of 45 test runs failed. That a 18% failure rate. It would be good to understand if all those failures are similar, what's causing them, and if they can easily be made more robust. -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to libreoffice in Ubuntu. https://bugs.launchpad.net/bugs/1785262 Title: arm64 autopkgtests are flaky Status in LibreOffice: Confirmed Status in libreoffice package in Ubuntu: Fix Released Bug description: [Verification and regression analysis is all handled via the main SRU bug of LP: #1913353] See http://autopkgtest.ubuntu.com/packages/libreoffice/cosmic/arm64. In the past 30 days, 8 out of 45 test runs failed. That a 18% failure rate. It would be good to understand if all those failures are similar, what's causing them, and if they can easily be made more robust. To manage notifications about this bug go to: https://bugs.launchpad.net/df-libreoffice/+bug/1785262/+subscriptions -- Mailing list: https://launchpad.net/~desktop-packages Post to : desktop-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~desktop-packages More help : https://help.launchpad.net/ListHelp
[Desktop-packages] [Bug 1785262] Re: arm64 autopkgtests are flaky
This bug was fixed in the package libreoffice - 1:7.1.1~rc2-2ubuntu1 --- libreoffice (1:7.1.1~rc2-2ubuntu1) hirsute; urgency=medium * Fix autopkgtest failures -- Rico Tzschichholz Mon, 01 Mar 2021 11:17:32 +0100 ** Changed in: libreoffice (Ubuntu) Status: Confirmed => Fix Released -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to libreoffice in Ubuntu. https://bugs.launchpad.net/bugs/1785262 Title: arm64 autopkgtests are flaky Status in LibreOffice: Confirmed Status in libreoffice package in Ubuntu: Fix Released Bug description: See http://autopkgtest.ubuntu.com/packages/libreoffice/cosmic/arm64. In the past 30 days, 8 out of 45 test runs failed. That a 18% failure rate. It would be good to understand if all those failures are similar, what's causing them, and if they can easily be made more robust. To manage notifications about this bug go to: https://bugs.launchpad.net/df-libreoffice/+bug/1785262/+subscriptions -- Mailing list: https://launchpad.net/~desktop-packages Post to : desktop-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~desktop-packages More help : https://help.launchpad.net/ListHelp
[Desktop-packages] [Bug 1785262] Re: arm64 autopkgtests are flaky
** Changed in: libreoffice (Ubuntu) Status: Fix Released => Confirmed -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to libreoffice in Ubuntu. https://bugs.launchpad.net/bugs/1785262 Title: arm64 autopkgtests are flaky Status in LibreOffice: Confirmed Status in libreoffice package in Ubuntu: Confirmed Bug description: See http://autopkgtest.ubuntu.com/packages/libreoffice/cosmic/arm64. In the past 30 days, 8 out of 45 test runs failed. That a 18% failure rate. It would be good to understand if all those failures are similar, what's causing them, and if they can easily be made more robust. To manage notifications about this bug go to: https://bugs.launchpad.net/df-libreoffice/+bug/1785262/+subscriptions -- Mailing list: https://launchpad.net/~desktop-packages Post to : desktop-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~desktop-packages More help : https://help.launchpad.net/ListHelp
[Desktop-packages] [Bug 1785262] Re: arm64 autopkgtests are flaky
This bug was fixed in the package libreoffice - 1:7.0.4~rc2-0ubuntu2 --- libreoffice (1:7.0.4~rc2-0ubuntu2) hirsute; urgency=medium * Revert "Re-enable coinmp (LP: #1905422)" until MIR is sorted out -- Rico Tzschichholz Mon, 14 Dec 2020 10:05:19 +0100 ** Changed in: libreoffice (Ubuntu) Status: Triaged => Fix Released -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to libreoffice in Ubuntu. https://bugs.launchpad.net/bugs/1785262 Title: arm64 autopkgtests are flaky Status in LibreOffice: Confirmed Status in libreoffice package in Ubuntu: Fix Released Bug description: See http://autopkgtest.ubuntu.com/packages/libreoffice/cosmic/arm64. In the past 30 days, 8 out of 45 test runs failed. That a 18% failure rate. It would be good to understand if all those failures are similar, what's causing them, and if they can easily be made more robust. To manage notifications about this bug go to: https://bugs.launchpad.net/df-libreoffice/+bug/1785262/+subscriptions -- Mailing list: https://launchpad.net/~desktop-packages Post to : desktop-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~desktop-packages More help : https://help.launchpad.net/ListHelp
Re: [Desktop-packages] [Bug 1785262] Re: arm64 autopkgtests are flaky
On Wed, Dec 09, 2020 at 10:46:52PM -, Rico Tzschichholz wrote: > This issue is not restricted to arm* but have been seen occasionally on amd64. > It effects the binary package build too, where these tests are part of "make > check". > > There are a lot tests disabled in > 'sw/qa/uitest/writer_dialogs/openDialogs.py' for similar reasons. > > So while it locks up at "test_21_.uno:EditGlossary (openDialogs.openDialogs) > ... " I would give that a try: > https://git.launchpad.net/~libreoffice/ubuntu/+source/libreoffice/commit/?id=0136e1c7a8f8823cbd0e5027ed203ee0d2f71262 Thanks for the bug reference - I had an unconfirmed theory that the test was deadlocking somewhere! I think Heather mentioned seeing this in other tests too - it would be good to grep the failing logs to see... seems like the string when it starts a test doesn't contain a \n, so it looks like: test_21_.uno:EditGlossary (openDialogs.openDialogs) ... autopkgtest [22:36:52]: ERROR: timed out on command "... If it's only this one, or only a small number of tests which can hang (on slow / loaded machines particularly it seems) always, then skipping those few would be a reasonable solution, indeed. But if this can apply to any of the uicheck tests, then I think I'm still in favour of using timeout(1) to paper over the deadlock, by setting our own timeout, lower than the autopkgtest one, and skipping if we hit it. Cheers, -- Iain Lane [ i...@orangesquash.org.uk ] Debian Developer [ la...@debian.org ] Ubuntu Developer [ la...@ubuntu.com ] -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to libreoffice in Ubuntu. https://bugs.launchpad.net/bugs/1785262 Title: arm64 autopkgtests are flaky Status in LibreOffice: Confirmed Status in libreoffice package in Ubuntu: Triaged Bug description: See http://autopkgtest.ubuntu.com/packages/libreoffice/cosmic/arm64. In the past 30 days, 8 out of 45 test runs failed. That a 18% failure rate. It would be good to understand if all those failures are similar, what's causing them, and if they can easily be made more robust. To manage notifications about this bug go to: https://bugs.launchpad.net/df-libreoffice/+bug/1785262/+subscriptions -- Mailing list: https://launchpad.net/~desktop-packages Post to : desktop-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~desktop-packages More help : https://help.launchpad.net/ListHelp
[Desktop-packages] [Bug 1785262] Re: arm64 autopkgtests are flaky
Launchpad has imported 3 comments from the remote bug at https://bugs.documentfoundation.org/show_bug.cgi?id=129838. If you reply to an imported comment from within Launchpad, your comment will be sent to the remote bug automatically. Read more about Launchpad's inter-bugtracker facilities at https://help.launchpad.net/InterBugTracking. On 2020-01-06T17:12:56+00:00 Michael-stahl wrote: Description: current master f1604675e71c67024887d12bf73ccb86ac05a7a3 the problem is: * thread 6 runs async call SwXTextDocument::release() which blocks to acquire SolarMutex * thread 1 holds SolarMutex, blocks on sync call on same aThreadId Steps to Reproduce: have only seen it once Actual Results: (gdb) thread apply all bt Thread 8 (Thread 0x7faa995c7700 (LWP 612145)): #0 0x7faabb116610 in __lll_lock_wait () at /lib64/libpthread.so.0 #1 0x7faabb10efe1 in pthread_mutex_lock () at /lib64/libpthread.so.0 #2 0x7faabb995ba6 in osl_acquireMutex(oslMutex) (pMutex=0x8b33b0) at /work/lo/master/sal/osl/unx/mutex.cxx:100 #3 0x7faab3a42867 in osl::Mutex::acquire() (this=0x8b85b8) at /work/lo/master/include/osl/mutex.hxx:57 #4 0x7faab43aa1ba in SvpSalYieldMutex::doAcquire(unsigned int) (this=0x8b85b0, nLockCount=1) at /work/lo/master/vcl/headless/svpinst.cxx:360 #5 0x7faaa90e8d6e in comphelper::SolarMutex::acquire(unsigned int) (this=0x8b85b0, nLockCount=1) at /work/lo/master/include/comphelper/solarmutex.hxx:86 #6 0x7faaa90e91a5 in osl::Guard::Guard(comphelper::SolarMutex&) (this=0x7faa995c5d80, t=...) at /work/lo/master/include/osl/mutex.hxx:135 #7 0x7faaa90e8dc3 in SolarMutexGuard::SolarMutexGuard() (this=0x7faa995c5d80) at /work/lo/master/include/vcl/svapp.hxx:1359 #8 0x7faaa9158476 in framework::OComponentAccess::createEnumeration() (this=0x5890aa0) at /work/lo/master/framework/source/helper/ocomponentaccess.cxx:55 #9 0x7faabb67ff28 in gcc3::callVirtualMethod(void*, unsigned int, void*, _typelib_TypeDescriptionReference*, bool, unsigned long*, unsigned int, unsigned long*, double*) (pThis=0x5890ac8, nVtableIndex=5, pRegisterReturn=0x7faa995c5f20, pReturnTypeRef=0x9f66b0, bSimpleReturn=false, pStack=0x7faa995c5f30, nStack=0, pGPR=0x7faa995c6060, pFPR=0x7faa995c6090) at /work/lo/master/bridges/source/cpp_uno/gcc3_linux_x86-64/callvirtualmethod.cxx:133 #10 0x7faabb67eb7b in cpp_call(bridges::cpp_uno::shared::UnoInterfaceProxy*, bridges::cpp_uno::shared::VtableSlot, typelib_TypeDescriptionReference*, sal_Int32, typelib_MethodParameter*, void*, void**, uno_Any**) (pThis=0x58914b0, aVtableSlot=..., pReturnTypeRef=0x9f66b0, nParams=0, pParams=0x0, pUnoReturn=0x588dce0, pUnoArgs=0x0, ppUnoExc=0x7faa995c6200) at /work/lo/master/bridges/source/cpp_uno/gcc3_linux_x86-64/uno2cpp.cxx:233 #11 0x7faabb67f596 in bridges::cpp_uno::shared::unoInterfaceProxyDispatch(uno_Interface*, typelib_TypeDescription const*, void*, void**, uno_Any**) (pUnoI=0x58914b0, pMemberDescr=0x9f8190, pReturn=0x588dce0, pArgs=0x0, ppException=0x7faa995c6200) at /work/lo/master/bridges/source/cpp_uno/gcc3_linux_x86-64/uno2cpp.cxx:413 #12 0x7faaa9f7a3d7 in binaryurp::IncomingRequest::execute_throw(binaryurp::BinaryAny*, std::__debug::vector >*) const (this=0x5891210, returnValue=0x7faa995c65c0, outArguments=0x7faa995c6640) at /work/lo/master/binaryurp/source/incomingrequest.cxx:236 #13 0x7faaa9f79280 in binaryurp::IncomingRequest::execute() const (this=0x5891210) at /work/lo/master/binaryurp/source/incomingrequest.cxx:79 #14 0x7faaa9f9de95 in binaryurp::(anonymous namespace)::request(void*) (pThreadSpecificData=0x5891210) at /work/lo/master/binaryurp/source/reader.cxx:85 #15 0x7faaba3c22de in cppu_threadpool::JobQueue::enter(long, bool) (this=0x5891c40, nDisposeId=92869344, bReturnWhenNoJob=true) at /work/lo/master/cppu/source/threadpool/jobqueue.cxx:107 #16 0x7faaba3c89ab in cppu_threadpool::ORequestThread::run() (this=0x58912e0) at /work/lo/master/cppu/source/threadpool/thread.cxx:165 #17 0x7faaba3c8eda in osl::threadFunc(void*) (param=0x58912f0) at /work/lo/master/include/osl/thread.hxx:185 #18 0x7faabb9acbfb in osl_thread_start_Impl(void*) (pData=0x20cb060) at /work/lo/master/sal/osl/unx/thread.cxx:256 #19 0x7faabb10c4e2 in start_thread () at /lib64/libpthread.so.0 #20 0x7faabb594693 in clone () at /lib64/libc.so.6 Thread 7 (Thread 0x7faa93c74700 (LWP 609090)): #0 0x7faabb112d45 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libpthread.so.0 #1 0x7faabb983130 in osl_waitCondition(oslCondition, TimeValue const*) (Condition=0x1a92660, pTimeout=0x0) at /work/lo/master/sal/osl/unx/conditn.cxx:231 #2 0x7faaaff0cb1c in osl::Condition::wait(TimeValue const*) (this=0x4fe2000, pTimeout=0x0) at /work/lo/master/include/osl/conditn.hxx:119 #3 0x7faaaff0768b in GrammarCheckingIterator::DequeueAndCheck() (this=0x4fe1e50) at
[Desktop-packages] [Bug 1785262] Re: arm64 autopkgtests are flaky
This issue is not restricted to arm* but have been seen occasionally on amd64. It effects the binary package build too, where these tests are part of "make check". There are a lot tests disabled in 'sw/qa/uitest/writer_dialogs/openDialogs.py' for similar reasons. So while it locks up at "test_21_.uno:EditGlossary (openDialogs.openDialogs) ... " I would give that a try: https://git.launchpad.net/~libreoffice/ubuntu/+source/libreoffice/commit/?id=0136e1c7a8f8823cbd0e5027ed203ee0d2f71262 -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to libreoffice in Ubuntu. https://bugs.launchpad.net/bugs/1785262 Title: arm64 autopkgtests are flaky Status in LibreOffice: Unknown Status in libreoffice package in Ubuntu: Triaged Bug description: See http://autopkgtest.ubuntu.com/packages/libreoffice/cosmic/arm64. In the past 30 days, 8 out of 45 test runs failed. That a 18% failure rate. It would be good to understand if all those failures are similar, what's causing them, and if they can easily be made more robust. To manage notifications about this bug go to: https://bugs.launchpad.net/df-libreoffice/+bug/1785262/+subscriptions -- Mailing list: https://launchpad.net/~desktop-packages Post to : desktop-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~desktop-packages More help : https://help.launchpad.net/ListHelp
[Desktop-packages] [Bug 1785262] Re: arm64 autopkgtests are flaky
** Bug watch added: Document Foundation Bugzilla #129838 https://bugs.documentfoundation.org/show_bug.cgi?id=129838 ** Also affects: df-libreoffice via https://bugs.documentfoundation.org/show_bug.cgi?id=129838 Importance: Unknown Status: Unknown -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to libreoffice in Ubuntu. https://bugs.launchpad.net/bugs/1785262 Title: arm64 autopkgtests are flaky Status in LibreOffice: Unknown Status in libreoffice package in Ubuntu: Triaged Bug description: See http://autopkgtest.ubuntu.com/packages/libreoffice/cosmic/arm64. In the past 30 days, 8 out of 45 test runs failed. That a 18% failure rate. It would be good to understand if all those failures are similar, what's causing them, and if they can easily be made more robust. To manage notifications about this bug go to: https://bugs.launchpad.net/df-libreoffice/+bug/1785262/+subscriptions -- Mailing list: https://launchpad.net/~desktop-packages Post to : desktop-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~desktop-packages More help : https://help.launchpad.net/ListHelp
[Desktop-packages] [Bug 1785262] Re: arm64 autopkgtests are flaky
After some more investigation, here are the main points that make this issue a tricky one to debug: 1. The test failure is not isolated. While the uicheck-sw test is the culprit, it is just an umbrella for many other tests. And it's those smaller tests that the failure occurs all over. This makes it hard to just "fix the tests". 2. I have not been able to reproduce this locally on a rpi4. Laney pointed out that perhaps having multiple containers running the same test might generate the load needed (assuming that plays a key role in the failures seen on LP) and I'll certainly give this a shot. 3. Currently if you look at the recent hirsute armhf and arm64 LO test results, you'll see that the rate of failure is way down (about 1/10 times). But just a couple of weeks ago the failure rate was much higher. This suggests that it's load on the nodes causing the problem because that's when the infra was quite busy. 4. Marking the test simply as "flaky" won't work because it's actually autopkgtest killing the process The approach going forward is to try and be clever about letting this uicheck-sw test timeout and not be considered a failure: https://git.launchpad.net/~hellsworth/+git/libreoffice/commit/?h=ubuntu-hirsute-7.0-workaround-arm-flakiness=63170c4e1824dcee45b670738ff47a6174a991a8 A test build on both armhf and arm64 to try out these changes: https://launchpad.net/~hellsworth/+archive/ubuntu/lo-arm-flaky-testing -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to libreoffice in Ubuntu. https://bugs.launchpad.net/bugs/1785262 Title: arm64 autopkgtests are flaky Status in libreoffice package in Ubuntu: Triaged Bug description: See http://autopkgtest.ubuntu.com/packages/libreoffice/cosmic/arm64. In the past 30 days, 8 out of 45 test runs failed. That a 18% failure rate. It would be good to understand if all those failures are similar, what's causing them, and if they can easily be made more robust. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libreoffice/+bug/1785262/+subscriptions -- Mailing list: https://launchpad.net/~desktop-packages Post to : desktop-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~desktop-packages More help : https://help.launchpad.net/ListHelp
[Desktop-packages] [Bug 1785262] Re: arm64 autopkgtests are flaky
> That would seem to rule out FTBFS for example too. We have been rls-not-fixing build failures in the past as well yes, especially in non lts series -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to libreoffice in Ubuntu. https://bugs.launchpad.net/bugs/1785262 Title: arm64 autopkgtests are flaky Status in libreoffice package in Ubuntu: Triaged Bug description: See http://autopkgtest.ubuntu.com/packages/libreoffice/cosmic/arm64. In the past 30 days, 8 out of 45 test runs failed. That a 18% failure rate. It would be good to understand if all those failures are similar, what's causing them, and if they can easily be made more robust. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libreoffice/+bug/1785262/+subscriptions -- Mailing list: https://launchpad.net/~desktop-packages Post to : desktop-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~desktop-packages More help : https://help.launchpad.net/ListHelp
[Desktop-packages] [Bug 1785262] Re: arm64 autopkgtests are flaky
> Would work for me, can we mark flaky-on-armhf only (I'd not be aware)? The only way I could think to do this would be to duplicate the test and use "Architecture:" to run the "Restrictions: flaky" one on armhf arm64 only and the other one otherwise. In this case the failure is a *timeout* though so I'm not sure that Restrictions: flaky works since it is being killed from the outside by autopkgtest itself (the test runner). That would need to be tested to make sure it does. You could do that by making a simple (in another package, doesn't have to be libreoffice) which has "Restricitons: flaky" and runs something like "sleep infinity". See what happens when autopkgtest times out the test - you can pass --timeout-test to reduce this value for testing. Onto the test itself a bit. When the "uicheck-sw" test passes, it takes in the region of 20-25 minutes. Picking two recent results: armhf: autopkgtest [05:43:20]: test uicheck-sw: [--- autopkgtest [06:03:54]: test uicheck-sw: ---] autopkgtest [06:03:58]: test uicheck-sw: - - - - - - - - - - results - - - - - - - - - - uicheck-sw PASS arm64: autopkgtest [02:52:01]: test uicheck-sw: [--- autopkgtest [03:05:33]: test uicheck-sw: ---] autopkgtest [03:05:34]: test uicheck-sw: - - - - - - - - - - results - - - - - - - - - - uicheck-sw PASS but the timeout we have (this is a "large" test, it runs with higher timeouts than we use by default) is after over 5 *hours* 30 minutes. It's clearly wildly variable, that's more than 10× as much time and it still timed out. If this is to do with the test runners being "loaded" (slower) vs "unloaded" (faster), then it could be a race condition that happens when the test runs slowly that causes it to get stuck? I'm not sure if you want to go down the route of investigating the actual failure, though, but it does feel to me like this "shouldn't" be failing. We're not usually close to the limits and just being tipped over them sometimes, does look like the test is actually getting stuck. But it would require someone to actually look at the tests themselves to determine that. Clearly having the tests working would be ideal; we want assurance that Libreoffice works on armhf and arm64. In the absence of this, since we're just mashing the retry button and not actually trying to fix them (ATM), making this stop blocking us one way or another would be better than the status quo. -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to libreoffice in Ubuntu. https://bugs.launchpad.net/bugs/1785262 Title: arm64 autopkgtests are flaky Status in libreoffice package in Ubuntu: Triaged Bug description: See http://autopkgtest.ubuntu.com/packages/libreoffice/cosmic/arm64. In the past 30 days, 8 out of 45 test runs failed. That a 18% failure rate. It would be good to understand if all those failures are similar, what's causing them, and if they can easily be made more robust. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libreoffice/+bug/1785262/+subscriptions -- Mailing list: https://launchpad.net/~desktop-packages Post to : desktop-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~desktop-packages More help : https://help.launchpad.net/ListHelp
[Desktop-packages] [Bug 1785262] Re: arm64 autopkgtests are flaky
> we don't believe it's a release issue It would be good to elaborate when making such comments. It's a drag on the archive in general when we have flaky tests, and even more so when they take many hours and potentially multiple retries to succeed. What's the criteria you're using here? Are we saying that "release" is only for user facing issues and bugs which have a negative impact on development of Ubuntu itself aren't to go through this process? If so, I think the team should come up with a way of being able to raise such issues. That would seem to rule out FTBFS for example too. -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to libreoffice in Ubuntu. https://bugs.launchpad.net/bugs/1785262 Title: arm64 autopkgtests are flaky Status in libreoffice package in Ubuntu: Triaged Bug description: See http://autopkgtest.ubuntu.com/packages/libreoffice/cosmic/arm64. In the past 30 days, 8 out of 45 test runs failed. That a 18% failure rate. It would be good to understand if all those failures are similar, what's causing them, and if they can easily be made more robust. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libreoffice/+bug/1785262/+subscriptions -- Mailing list: https://launchpad.net/~desktop-packages Post to : desktop-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~desktop-packages More help : https://help.launchpad.net/ListHelp
[Desktop-packages] [Bug 1785262] Re: arm64 autopkgtests are flaky
Tagging notfixing, the bug is assigned and we will try to get it fixed rather sooner than later but we don't believe it's a release issue ** Tags removed: rls-hh-incoming ** Tags added: rls-hh-notfixing -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to libreoffice in Ubuntu. https://bugs.launchpad.net/bugs/1785262 Title: arm64 autopkgtests are flaky Status in libreoffice package in Ubuntu: Triaged Bug description: See http://autopkgtest.ubuntu.com/packages/libreoffice/cosmic/arm64. In the past 30 days, 8 out of 45 test runs failed. That a 18% failure rate. It would be good to understand if all those failures are similar, what's causing them, and if they can easily be made more robust. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libreoffice/+bug/1785262/+subscriptions -- Mailing list: https://launchpad.net/~desktop-packages Post to : desktop-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~desktop-packages More help : https://help.launchpad.net/ListHelp
[Desktop-packages] [Bug 1785262] Re: arm64 autopkgtests are flaky
** Changed in: libreoffice (Ubuntu) Assignee: (unassigned) => Heather Ellsworth (hellsworth) -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to libreoffice in Ubuntu. https://bugs.launchpad.net/bugs/1785262 Title: arm64 autopkgtests are flaky Status in libreoffice package in Ubuntu: Triaged Bug description: See http://autopkgtest.ubuntu.com/packages/libreoffice/cosmic/arm64. In the past 30 days, 8 out of 45 test runs failed. That a 18% failure rate. It would be good to understand if all those failures are similar, what's causing them, and if they can easily be made more robust. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libreoffice/+bug/1785262/+subscriptions -- Mailing list: https://launchpad.net/~desktop-packages Post to : desktop-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~desktop-packages More help : https://help.launchpad.net/ListHelp
Re: [Desktop-packages] [Bug 1785262] Re: arm64 autopkgtests are flaky
On Fri, Nov 20, 2020 at 9:00 PM Heather Ellsworth <1785...@bugs.launchpad.net> wrote: > > I agree that the uicheck-sw test has increased in flakiness. How about > instead of removing the test or sinking the resources into actually > fixing them right now, I can mark them flaky so that at least a failed > test looks like a skipped test: Would work for me, can we mark flaky-on-armhf only (I'd not be aware)? -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to libreoffice in Ubuntu. https://bugs.launchpad.net/bugs/1785262 Title: arm64 autopkgtests are flaky Status in libreoffice package in Ubuntu: Triaged Bug description: See http://autopkgtest.ubuntu.com/packages/libreoffice/cosmic/arm64. In the past 30 days, 8 out of 45 test runs failed. That a 18% failure rate. It would be good to understand if all those failures are similar, what's causing them, and if they can easily be made more robust. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libreoffice/+bug/1785262/+subscriptions -- Mailing list: https://launchpad.net/~desktop-packages Post to : desktop-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~desktop-packages More help : https://help.launchpad.net/ListHelp
[Desktop-packages] [Bug 1785262] Re: arm64 autopkgtests are flaky
I agree that the uicheck-sw test has increased in flakiness. How about instead of removing the test or sinking the resources into actually fixing them right now, I can mark them flaky so that at least a failed test looks like a skipped test: https://salsa.debian.org/ci-team/autopkgtest/blob/master/doc/README .package-tests.rst -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to libreoffice in Ubuntu. https://bugs.launchpad.net/bugs/1785262 Title: arm64 autopkgtests are flaky Status in libreoffice package in Ubuntu: Triaged Bug description: See http://autopkgtest.ubuntu.com/packages/libreoffice/cosmic/arm64. In the past 30 days, 8 out of 45 test runs failed. That a 18% failure rate. It would be good to understand if all those failures are similar, what's causing them, and if they can easily be made more robust. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libreoffice/+bug/1785262/+subscriptions -- Mailing list: https://launchpad.net/~desktop-packages Post to : desktop-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~desktop-packages More help : https://help.launchpad.net/ListHelp
[Desktop-packages] [Bug 1785262] Re: arm64 autopkgtests are flaky
This recently showed up often enough that I bothered to find this bug :-/ I think it degraded again into a state where we should consider: IRC: [13:16] Was there any hope to not expect this to be a retry-monster forever? [13:17] The desktop team should be asked if they can work on making that less flaky [13:17] I don't think ignoring the tests there is a good answer [13:18] Reducing the coverage would be preferable if that gets it to be more stable, for example, assuming the flakiness can't be easily fixed I have a simple test stats gatherer [1] that also shows the subtests which helps a lot in those cases to identify potential candidates to skip. I attached the output, best viewed in monospace AFAICS up until recently we had <15% on all architectures. But in Hirsute on arm* I see a bump to 22% of the subtest "uicheck-sw". Could be bad-luck / noise - but maybe it is worth considering to skip that on arm? [1]: https://git.launchpad.net/~ubuntu-server/ubuntu- helpers/tree/cpaelzer/check-autopkgtest-stats.sh ** Attachment added: "recent test statistics" https://bugs.launchpad.net/ubuntu/+source/libreoffice/+bug/1785262/+attachment/5435973/+files/libreoffice-test-stats.txt -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to libreoffice in Ubuntu. https://bugs.launchpad.net/bugs/1785262 Title: arm64 autopkgtests are flaky Status in libreoffice package in Ubuntu: Triaged Bug description: See http://autopkgtest.ubuntu.com/packages/libreoffice/cosmic/arm64. In the past 30 days, 8 out of 45 test runs failed. That a 18% failure rate. It would be good to understand if all those failures are similar, what's causing them, and if they can easily be made more robust. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libreoffice/+bug/1785262/+subscriptions -- Mailing list: https://launchpad.net/~desktop-packages Post to : desktop-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~desktop-packages More help : https://help.launchpad.net/ListHelp
[Desktop-packages] [Bug 1785262] Re: arm64 autopkgtests are flaky
Christian raised this with us just now - I think we should consider taking a look at this. Looking at http://autopkgtest.ubuntu.com/packages/libreoffice/hirsute/arm64 http://autopkgtest.ubuntu.com/packages/libreoffice/hirsute/armhf the tests are basically OK but one of the uicheck-* ones ends up timing out often enough that it can be described as flaky. When this happens it causes a problem for other migrations, doubly so as the runs take many hours to complete whether they fail or not, which slows down migration speed. If it's not possible to fix the timeouts (feels like it might be hard or take a while) then disabling these subtests (only) on arm would be an option - stable tests with less coverage are better than flaky ones with more. We had https://git.launchpad.net/~libreoffice/ubuntu/+source/libreoffice/commit/?id=baf5a581da7ce3bb27a5b8ed44fadeb6e245e1ff in the past which might be a basis for what to do now, but that would want to be extended to arm64 too. -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to libreoffice in Ubuntu. https://bugs.launchpad.net/bugs/1785262 Title: arm64 autopkgtests are flaky Status in libreoffice package in Ubuntu: Triaged Bug description: See http://autopkgtest.ubuntu.com/packages/libreoffice/cosmic/arm64. In the past 30 days, 8 out of 45 test runs failed. That a 18% failure rate. It would be good to understand if all those failures are similar, what's causing them, and if they can easily be made more robust. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libreoffice/+bug/1785262/+subscriptions -- Mailing list: https://launchpad.net/~desktop-packages Post to : desktop-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~desktop-packages More help : https://help.launchpad.net/ListHelp
[Desktop-packages] [Bug 1785262] Re: arm64 autopkgtests are flaky
** Tags added: rls-hh-incoming -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to libreoffice in Ubuntu. https://bugs.launchpad.net/bugs/1785262 Title: arm64 autopkgtests are flaky Status in libreoffice package in Ubuntu: Triaged Bug description: See http://autopkgtest.ubuntu.com/packages/libreoffice/cosmic/arm64. In the past 30 days, 8 out of 45 test runs failed. That a 18% failure rate. It would be good to understand if all those failures are similar, what's causing them, and if they can easily be made more robust. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libreoffice/+bug/1785262/+subscriptions -- Mailing list: https://launchpad.net/~desktop-packages Post to : desktop-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~desktop-packages More help : https://help.launchpad.net/ListHelp
[Desktop-packages] [Bug 1785262] Re: arm64 autopkgtests are flaky
** Changed in: libreoffice (Ubuntu) Assignee: Olivier Tilloy (osomon) => (unassigned) -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to libreoffice in Ubuntu. https://bugs.launchpad.net/bugs/1785262 Title: arm64 autopkgtests are flaky Status in libreoffice package in Ubuntu: Triaged Bug description: See http://autopkgtest.ubuntu.com/packages/libreoffice/cosmic/arm64. In the past 30 days, 8 out of 45 test runs failed. That a 18% failure rate. It would be good to understand if all those failures are similar, what's causing them, and if they can easily be made more robust. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libreoffice/+bug/1785262/+subscriptions -- Mailing list: https://launchpad.net/~desktop-packages Post to : desktop-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~desktop-packages More help : https://help.launchpad.net/ListHelp
[Desktop-packages] [Bug 1785262] Re: arm64 autopkgtests are flaky
1 failure out of 57 test runs in the past 30 days, that's a 1.75% failure rate. -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to libreoffice in Ubuntu. https://bugs.launchpad.net/bugs/1785262 Title: arm64 autopkgtests are flaky Status in libreoffice package in Ubuntu: Triaged Bug description: See http://autopkgtest.ubuntu.com/packages/libreoffice/cosmic/arm64. In the past 30 days, 8 out of 45 test runs failed. That a 18% failure rate. It would be good to understand if all those failures are similar, what's causing them, and if they can easily be made more robust. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libreoffice/+bug/1785262/+subscriptions -- Mailing list: https://launchpad.net/~desktop-packages Post to : desktop-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~desktop-packages More help : https://help.launchpad.net/ListHelp
[Desktop-packages] [Bug 1785262] Re: arm64 autopkgtests are flaky
3 failures out of 54 test runs in the past 30 days, that's a 5.6% failure rate. -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to libreoffice in Ubuntu. https://bugs.launchpad.net/bugs/1785262 Title: arm64 autopkgtests are flaky Status in libreoffice package in Ubuntu: Triaged Bug description: See http://autopkgtest.ubuntu.com/packages/libreoffice/cosmic/arm64. In the past 30 days, 8 out of 45 test runs failed. That a 18% failure rate. It would be good to understand if all those failures are similar, what's causing them, and if they can easily be made more robust. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libreoffice/+bug/1785262/+subscriptions -- Mailing list: https://launchpad.net/~desktop-packages Post to : desktop-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~desktop-packages More help : https://help.launchpad.net/ListHelp
