On Tue, Jul 28, 2026 at 3:04 PM Jakub Wartak <[email protected]> wrote: > > On Tue, Jul 28, 2026 at 7:15 AM Xuneng Zhou <[email protected]> wrote: > > > > Hi, > > > > On Tue, Jul 28, 2026 at 12:00 PM Alexander Lakhin <[email protected]> > > wrote: > > > > > > Hello Aleksander and Michael, > > > > > > 28.07.2026 05:24, Michael Paquier пишет: > > > > > > On Mon, Jul 27, 2026 at 04:36:24PM +0300, Aleksander Alekseev wrote: > > > > > > AI suggests the new behavior is caused by the "io_uring: use region api > > > for SQ/CQ" change landed in the kernel version 6.14. > > > > > > No idea about the AI argument, but.. > > > > > > Two cents: I'm on Ubuntu 24.04 with Linux 6.14.0-37-generic and > > > liburing 2.5 and the tests pass fine for me. > > > > > > You are not alone. Various buildfarm members are also turning red > > > with random failures across multiple tests because of this issue. > > > > > > > > > I think, I've found the difference -- on Debian I'm seeing: > > > user@debian:~$ ulimit -l > > > 8192 > > > vs > > > vagrant@ubuntu:~$ ulimit -l > > > 2044616 > > > > > > Thus, with "ulimit -l 8192" set on Ubuntu 24.04, I've got the aio > > > failures there as well: > > > Linux ubuntu 6.14.0-37-generic #37~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu > > > Nov 20 10:25:38 UTC 2 x86_64 x86_64 x86_64 GNU/Linux > > > > > > $ meson test test_aio_* -j 30 > > > ... > > > Ok: 101 > > > Expected Fail: 0 > > > Fail: 19 > > > Unexpected Pass: 0 > > > > > > Also reproduced on Fedora, with: > > > Linux localhost.localdomain 6.17.10-100.fc41.x86_64 #1 SMP > > > PREEMPT_DYNAMIC Mon Dec 1 16:10:21 UTC 2025 x86_64 GNU/Linux > > > $ ulimit -l > > > 8192 > > > (default) > > > > Out of interest, I let Sol do some investigations for this issue. The > > conclusion it reached seems to align with that of the AI tool used by > > Alexander. However, I lacked the expertise to fully review/confirm it. > > Not sure whether it is acceptable to post a 'could-be-sloppy' report > > here. > > > Interesting. To me that output makes a lot of sense. We are having plenty of > io_urings allocated (one for each connection up to max_connections), they are > small, but the tests allocate many separate clusters too and all of this hits > the ulimit of max locked memory. I think the 8MB is the norm (e.g. > systemd-system.conf(5) says "DefaultLimitMEMLOCK= defaults to 8MB and there > can be per-service LimitMEMLOCK too), but Ubuntu seems to be way less > conservative and more open as it's mostly for users with X11/audio daemons > (pipewire) and stuff like that. > > Sadly soft:hard limit is the same, so we won't be able to bump it like in [1] > > Dunno what to do, it's somewhat silent(?) kernel change: reduce parallelism of > the tests maybe with even lower max_connections, bump DefaultLimitMEMLOCK ordo > it in per-service (but dunno how's buildfarm env looks like), bring back older > kernels (but what about security fixes?) > > If that AI report is to believed we are going to need to update docs as it > will be hit in the wild too: > > - as max_connections=1k + io_uring won't possible on newest kernels (but I > haven't verified the math behind it) > > - on non-standard > 4kB page systems (ARM?) same thing might be happening > (2 pages won't 8192 there, but much more hitting earlier the ENOMEM > > -J. > > [1] - > https://www.postgresql.org/message-id/DJS6KELUH3T0.2DGCBB9I190X%40jeltef.nl
I've started to look into this issue seriously. The upstream has landed a fix on its side citing this thread[1] [2]. Let's see what we can do on our side. [1] https://github.com/torvalds/linux/commit/b59cabf9b4acb65f3f9968087ca0e18010d85076 [2] https://lore.kernel.org/io-uring/[email protected]/T/#u -- Regards, Xuneng Zhou HighGo Software Co., Ltd.
