Will also fix #1979695 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1970634
Title: FTBFS: mariadb fails to start due to low MEMLOCK limit Status in mariadb-10.6 package in Ubuntu: Fix Released Status in systemd package in Ubuntu: Confirmed Status in mariadb-10.6 source package in Jammy: Fix Committed Status in systemd source package in Jammy: New Bug description: [Impact] Jammy's MariaDB will fail to build, and also fail to start, if the underlying kernel is 5.4.x (focal's) and if it's running in an unprivileged container (lxd, docker). It will also fail to build in launchpad builders. Common scenarios where this combination exists is a focal host, running an unprivileged jammy container (lxd or docker), or even a chroot (the launchpad builders). Jammy's MariaDB was built with io_uring support, and it tries to enable it at runtime if it deems it's running on a supported kernel. There is a range of kernel versions it checks, but of interest for this SRU, the Focal (5.4.x) kernel is inside this range and io_uring will be enabled. The jammy kernel (5.15) is not, so io_uring will not be enabled there, and thus the bug is not manifested in that case. If io_uring is enabled, a higher MEMLOCK limit is required than what is set by default in focal or jammy (64Mb is what we get, 256Mb or more is required). The systemd unit file for mariadb tries to raise this limit, but in an unprivileged container this won't work. MariaDB has checks in place to catch when MEMLOCK is too low, in which case it will not use io_uring, but these checks are failing because of the LTO build flags that were used in the jammy build of mariadb. It's unclear if it's a bug in gcc or something else. There is more information in comment #1, comment #5 and later. The suggested fix here is to disable LTO for the jammy build. This has been done for kinetic already, and is also applied to the debian packaging (inside a distro-specific conditional). [Test Plan] The test plan is to launch mariadb in a jammy lxd container running on a focal host. lxc launch ubuntu:focal f --vm lxc shell f lxd init # just hit enter for all questions lxc launch ubuntu:jammy j lxc shell j ulimit -l # confirm it's less than 256 apt update && apt install mariadb-server -y After the installation, mariadb will not be running, and this can be checked with: systemctl status mariadb.service or journalctl -u mariadb.server You will see something like this: Jun 17 18:32:01 jammy-mariadb systemd[1]: Starting MariaDB 10.6.7 database server... Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] /usr/sbin/mariadbd (server 10.6.7-MariaDB-2ubuntu1) starting as process 1864 ... Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: The first data file './ibdata1' did not exist. A new tablespace will be created! Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Using transactional memory Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Number of pools: 1 Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 2022-06-17 18:32:01 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd (262144 bytes required) Jun 17 18:32:01 jammy-mariadb mariadbd[1864]: 220617 18:32:01 [ERROR] mysqld got signal 6 ; And a crash dump. With the fixed version, the service will be running normally after installation. [Where problems could occur] The proposed fix is not a surgical strike. It's unfortunate that we didn't get to the bottom of why LTO is causing this behavior. Reverting it is still the quickest and less risky change at the moment, though. This gets us on par with upstream binary builds, and debian builds, and these also have wide test coverage and ample user base. The other regression possibility is that this is a rebuild of mariadb in the current jammy environment, and the package that is currently in jammy was built on March 10th, 2022. Most likely the reverse dependencies were updated in jammy since then. It's unclear how 10.6.7-2ubuntu1 migrated in jammy. I checked build logs and dep8 logs, and can't tell why the tests passed. At least the build log in jammy shows the host kernel was 5.4.x, so it should have been affected. My only explanation is that at that time, the MEMLOCK limit was higher in that environment for some reason, and didn't trigger this bug. Then at some point later, launchpad builders changed, and MEMLOCK was reduced to 64Mb. https://irclogs.ubuntu.com/2022/04/28/%23ubuntu-devel.html#t15:00 has some discussion about it. [Other Info] Not at this time. [Original Description] <rbasak> ahasenack: IIRC, originally Launchpad was FTBFSing on mariadb that included io_uring support because upstream were doing a build time test for io_uring (and I think still are), which is wrong because it should be done at runtime since the lack of io_uring availablity at build time doesn't tell us about its availablity at runtime. <rbasak> But then the Launchpad builders got updated to a newer release and therefore a newer kernel that supported it. <rbasak> AIUI, that's how we ended up with a successful build in the Jammy release pocket (of 10.6). <ahasenack> I think the lp builders are using the focal hwe kernel <ahasenack> 5.4.0-something <ahasenack> let me check that build log <rbasak> But then something changed that caused this current FTBFS, and I haven't tracked down what that is. <ahasenack> hm, both are 10.6.7 <ahasenack> release and proposed <rbasak> What puzzles me is that if the root cause is a memlock rlimit issue then why did it work before? <rbasak> So since there's a contradiction somewhere, maybe one or more of my "facts" above is wrong. <ahasenack> this is the current failure <ahasenack> 2022-04-14 8:11:49 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd (262144 bytes required) <ahasenack> and ulimit -l confirms that the limit is lower <ahasenack> Max locked memory 65536 65536 bytes <ahasenack> just 64kbytes <rbasak> Yeah but then how did the release pocket build work? <ahasenack> either the limit was different back then <ahasenack> or ... stuff To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mariadb-10.6/+bug/1970634/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
