Bug#1059904: mariadb autopkgtest "upstream" is flaky, time sink
New 1:10.116-2 uploaded, looking good so far at https://tracker.debian.org/pkg/mariadb: autopkgtest for mariadb/1:10.11.6-2: amd64: Pass, arm64: Pass, armel: Pass, armhf: Pass, i386: Pass, ppc64el: Test in progress, s390x: Pass
Bug#1059904: mariadb autopkgtest "upstream" is flaky, time sink
## arm64 All green at https://ci.debian.net/packages/m/mariadb/testing/arm64/ Latest log has "Completed: All 1049 tests were successful." ## i386 Likewise, all green at https://ci.debian.net/packages/m/mariadb/testing/i386/ Latest log has "Completed: All 1047 tests were successful.". ## armel Flaky at https://ci.debian.net/packages/m/mariadb/testing/armel/ Last passing a couple days ago: Completed: All 1047 tests were successful. All failing ones on main.subselect* for which I have a bugfix. ## armhf Flaky at https://ci.debian.net/packages/m/mariadb/testing/armhf/ Latest passed: Completed: All 1046 tests were successful. Likewise to armel, failures on main.subselect* ## s390x Flaky at https://ci.debian.net/packages/m/mariadb/testing/s390x/ Last passed a couple days ago: All 1047 tests were successful. Likewise to armel, failures on main.subselect* ## amd64 Finally, amd64 flaky at https://ci.debian.net/packages/m/mariadb/testing/amd64/ Last passed a couple days ago: Completed: All 1049 tests were successful. All failures on subselect as well, which is an upstream regression in 10.11.6: https://jira.mariadb.org/browse/MDEV-32843 Fix is pending on https://salsa.debian.org/otto/mariadb-server/-/commits/dev-otto but still need to finalize a bit before upload. Summary: the number of tests is not flaky on passed runs. The number of attempted test runs is fluctuating only when failures happend and tests re-run.
Bug#1059904: mariadb autopkgtest "upstream" is flaky, time sink
Summary of current status of https://ci.debian.net/packages/m/mariadb/ ## ppc64el Last passed in https://ci.debian.net/packages/m/mariadb/testing/ppc64el/40870773/ where 1048 tests were successful: MariaDB: 1:10.6.11-1 kernel: Linux 6.5.0-0.deb12.4-powerpc64le The log above includes 4 variants of main.innodb_ext_key passing. Later we see: Completed: Failed 8/1053 tests, 99.24% were successful. Failing test(s): main.innodb_ext_key main.stat_tables_innodb main.xa_prepared_binlog_off Completed: Failed 11/1055 tests, 98.96% were successful. Failing test(s): main.innodb_ext_key main.subselect_sj2_jcl6 Completed: Failed 3/1050 tests, 99.71% were successful. Failing test(s): main.innodb_ext_key Completed: Failed 2/1049 tests, 99.81% were successful. Failing test(s): main.innodb_ext_key Completed: Failed 3/1050 tests, 99.71% were successful. Failing test(s): main.xa_prepared_binlog_off Completed: Failed 10/1054 tests, 99.05% were successful. Failing test(s): main.innodb_ext_key main.subselect MariaDB: 1:10.6.11-1 kernel: Linux 6.5.0-0.deb12.4-powerpc64le In https://ci.debian.net/packages/m/mariadb/testing/ppc64el/40894691/ we see: Completed: All 1048 tests were successful. Errors/warnings were found in logfiles during server shutdown after running the following sequence(s) of tests: main.long_unique_bugs 183 tests were skipped, 76 by the test itself. The test emitted a MariaDB crash: main.long_unique_bugs 'innodb' w1 [ pass ] 324 14:51:15 [ERROR] mysqld got signal 11 ; Attempting backtrace. You can use the following information to find out Since 2023-12-18 16:13:06 UTC the test has failed to even start. The ppc64el logs have crashes and most likely due to https://jira.mariadb.org/browse/MDEV-30728, in particular the main.innodb_ext_key one.
Bug#1059904: mariadb autopkgtest "upstream" is flaky, time sink
I am working on a 1:10.11.6-2 upload to fix these. If I can make them reliable, I will temporarily disable the test suite.
Bug#1059904: mariadb autopkgtest "upstream" is flaky, time sink
Source: mariadb Version: 1:10.11.6-1 Severity: important Hi, the "upstream" autopkgtest appears to be flaky. On various test runs that I've inspected in the last few weeks, it often ended up with the following message, and a re-run seemingly passed without any other changes. > 356s Only 1242 of 1240 completed. ... > 356s Completed: Failed 3/1048 tests, 99.71% were successful. Somehow it would seem great if these numbers would actually add up. Also, from the log output I can see only *one* test marked as "[ fail ]". Even that one apparently gets retried automatically and then passes? But more importantly, if these tests cannot be made to pass or succeed consistently, please mark them as flaky, as right now this is just a waste of everybodys time. mariadb gets tested as part of the migration tests for so many other packages, it's not funny. Thanks, Chris
