Hello, I've identified the first bad commit using git bisect:
90bfb28d5fa8127a113a140c9791ea0b40ab156a is the first bad commit commit 90bfb28d5fa8127a113a140c9791ea0b40ab156a Author: Jens Axboe <[email protected]> Date: Tue Sep 10 08:57:04 2024 -0600 io_uring/rw: drop -EOPNOTSUPP check in __io_complete_rw_common() A recent change ensured that the necessary -EOPNOTSUPP -> -EAGAIN transformation happens inline on both the reader and writer side, and hence there's no need to check for both of these anymore on the completion handler side. Signed-off-by: Jens Axboe <[email protected]> io_uring/rw.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) Here is the git bisect log as well: git bisect start # status: waiting for both good and bad commits # good: [98f7e32f20d28ec452afb208f9cffc08448a2652] Linux 6.11 git bisect good 98f7e32f20d28ec452afb208f9cffc08448a2652 # status: waiting for bad commit, 1 good commit known # bad: [59b723cd2adbac2a34fc8e12c74ae26ae45bf230] Linux 6.12-rc6 git bisect bad 59b723cd2adbac2a34fc8e12c74ae26ae45bf230 # bad: [de848da12f752170c2ebe114804a985314fd5a6a] Merge tag 'drm-next- 2024-09-19' of https://gitlab.freedesktop.org/drm/kernel git bisect bad de848da12f752170c2ebe114804a985314fd5a6a # bad: [7b17f5ebd5fc5e9275eaa5af3d0771f2a7b01bbf] Merge tag 'soc-dt- 6.12' of git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc git bisect bad 7b17f5ebd5fc5e9275eaa5af3d0771f2a7b01bbf # good: [64dd3b6a79f0907d36de481b0f15fab323a53e5a] Merge tag 'for- linus-non-x86' of git://git.kernel.org/pub/scm/virt/kvm/kvm git bisect good 64dd3b6a79f0907d36de481b0f15fab323a53e5a # bad: [daa394f0f9d3cb002c72e2d3db99972e2ee42862] Merge tag 'core- debugobjects-2024-09-16' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip git bisect bad daa394f0f9d3cb002c72e2d3db99972e2ee42862 # good: [effdcd5275ed645f6e0f8e8ce690b97795722197] Merge tag 'affs-for- 6.12-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux git bisect good effdcd5275ed645f6e0f8e8ce690b97795722197 # bad: [26bb0d3f38a764b743a3ad5c8b6e5b5044d7ceb4] Merge tag 'for- 6.12/block-20240913' of git://git.kernel.dk/linux git bisect bad 26bb0d3f38a764b743a3ad5c8b6e5b5044d7ceb4 # bad: [3a4d319a8fb5a9bbdf5b31ef32841eb286b1dcc2] Merge tag 'for- 6.12/io_uring-20240913' of git://git.kernel.dk/linux git bisect bad 3a4d319a8fb5a9bbdf5b31ef32841eb286b1dcc2 # good: [df2825e98507d10cb037a308087ecd7cb3f6688d] btrfs: always pass readahead state to defrag git bisect good df2825e98507d10cb037a308087ecd7cb3f6688d # good: [69a3a0a45a2f72412c2ba31761cc9193bb746fef] Merge tag 'erofs- for-6.12-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs git bisect good 69a3a0a45a2f72412c2ba31761cc9193bb746fef # good: [ecd5c9b29643f383d39320e30d21b8615bd893da] io_uring/kbuf: add io_kbuf_commit() helper git bisect good ecd5c9b29643f383d39320e30d21b8615bd893da # good: [f011c9cf04c06f16b24f583d313d3c012e589e50] io_uring/sqpoll: do not allow pinning outside of cpuset git bisect good f011c9cf04c06f16b24f583d313d3c012e589e50 # bad: [84eacf177faa605853c58e5b1c0d9544b88c16fd] io_uring/io-wq: inherit cpuset of cgroup in io worker git bisect bad 84eacf177faa605853c58e5b1c0d9544b88c16fd # bad: [90bfb28d5fa8127a113a140c9791ea0b40ab156a] io_uring/rw: drop - EOPNOTSUPP check in __io_complete_rw_common() git bisect bad 90bfb28d5fa8127a113a140c9791ea0b40ab156a # good: [c0a9d496e0fece67db777bd48550376cf2960c47] io_uring/rw: treat - EOPNOTSUPP for IOCB_NOWAIT like -EAGAIN git bisect good c0a9d496e0fece67db777bd48550376cf2960c47 # first bad commit: [90bfb28d5fa8127a113a140c9791ea0b40ab156a] io_uring/rw: drop -EOPNOTSUPP check in __io_complete_rw_common() On Thu, 2025-10-02 at 16:25 +0200, Salvatore Bonaccorso wrote: > Control: tags -1 + moreinfo > > Hi Kevin, > > On Thu, Oct 02, 2025 at 02:28:48PM +0300, Kevin Lumik wrote: > > On Sat, 27 Sep 2025 16:09:08 +0200 Salvatore Bonaccorso > > <[email protected]> wrote: > > > - Can you temporarily test as well the upper kernels from > > > unstable > > > (6.16.8-1 right now, but soonish 6.16.9-1 as well) to see if > > > the > > > problem is triggerable there as well? > > > > I tested the upper kernels and was still able to reproduce the > > issue on > > both 6.16.8-1 and 6.16.9-1. > > Ok thanks for doing so, have just updated as well the metadata to > reflect that. > > > > Note: If we find the problem is prosent in 6.16.y then we need to > > > find > > > the breaking change between 6.1.y and 6.12.y. It might be asier > > > here > > > if we first make the range of potential upstream versions more > > > narrow, > > > so you might first do a "manual bisect" on Debian revisions (For > > > that > > > use the snapshot.debian.org service to retrieve linux-images > > > uploaded > > > to Debian. E.g. go down from 6.12 in "major version" steps until > > > we > > > have close enough range of broken/fixed upstream major versions. > > > Then > > > perform a normal bisect between the upstream versions. > > > > Using snapshot.debian.org I have determined that the breaking > > change was > > introduced between 6.11.10 (tested 6.11.10-1) and 6.12 (tested > > 6.12.3-1 > > and 6.12~rc6-1~exp1). > > > > Thank you for your assistance. Let me know what would be the next > > step. > > Okay so we have a good range to look at. The next steps would be to > bisect the changes. Actually it would be good to first test 6.11 > itself upstream. So the steps to do would be roughly the following, > which needs compiling and testing a few kernels: > > git clone > https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git > cd linux-stable > git checkout v6.11 > cp /boot/config-$(uname -r) .config > yes '' | make localmodconfig > make savedefconfig > mv defconfig arch/x86/configs/my_defconfig > > # test 6.11 to ensure this is "good" > make my_defconfig > make -j $(nproc) bindeb-pkg > ... install the resulting .deb package and confirm it > successfully boots / problem does not exist > > # test 6.12-rc6 to ensure this is "bad" > git checkout v6.12-rc6 > make my_defconfig > make -j $(nproc) bindeb-pkg > ... install the resulting .deb package and confirm it fails to > boot / problem exists > > With that confirmed, the bisection can start: > > git bisect start > git bisect good v6.11 > git bisect bad v6.12-rc6 > > In each bisection step git checks out a state between the oldest > known-bad and the newest known-good commit. In each step test using: > > make my_defconfig > make -j $(nproc) bindeb-pkg > ... install, try to boot / verify if problem exists > > and if the problem is hit run: > > git bisect bad > > and if the problem doesn't trigger run: > > git bisect good > > . Please pay attention to always select the just built kernel for > booting, it won't always be the default kernel picked up by grub. > > Iterate until git announces to have identified the first bad commit. > > Then provide the output of > > git bisect log > > In the course of the bisection you might have to uninstall previous > kernels again to not exhaust the disk space in /boot. Also in the end > uninstall all self-built kernels again. > > Hope this helps so far? > > Regards, > Salvatore
