[Kernel-packages] [Bug 2044657] Re: zfs block cloning file system corruption
Forgot to mention you also need coreutils-9.0 or later, or some other program that uses lseek SEEK_HOLE/SEEK_DATA as that is the culprit for the bug. Essentially it errorneously reports holes in files that are still dirty buffers. I had a local copy of "cp" from coreutils/sid compiled for jammy. So the primary trigger of the original report was always coreutils, but obviously any software can use hole detection features of lseek and run into it. I personally haven't noticed any data corruption. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-hwe-6.5 in Ubuntu. https://bugs.launchpad.net/bugs/2044657 Title: zfs block cloning file system corruption Status in linux-hwe-6.2 package in Ubuntu: Confirmed Status in linux-hwe-6.5 package in Ubuntu: Confirmed Status in zfs-linux package in Ubuntu: Confirmed Bug description: OpenZFS 2.2 reportedly has a bug where block cloning might lead to file system corruption and data loss. This was fixed in OpenZFS 2.2.1. Original bug report: https://github.com/openzfs/zfs/issues/15526 and 2.2.1 release notes: https://github.com/openzfs/zfs/releases/tag/zfs-2.2.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-hwe-6.2/+bug/2044657/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2044657] Re: zfs block cloning file system corruption
Reproducible on Ubuntu 22.04 LTS w/ linux-hwe-6.2 (zfs-2.1.9) using the NixOS test suite: [zhammer::647] checking 1 files at iteration 0 [zhammer::647] zhammer_647_0 differed from zhammer_647_576! [zhammer::647] Hexdump diff follows --- zhammer_647_0.hex 2023-11-30 15:37:43.887596987 + +++ zhammer_647_576.hex 2023-11-30 15:37:43.891596970 + @@ -1,3 +1,3 @@ - ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff || + 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 || * 4000 [zhammer::639] checking 1 files at iteration 0 [zhammer::631] checking 1 files at iteration 0 [zhammer::635] checking 1 files at iteration 0 [zhammer::677] checking 1 files at iteration 0 parallel: This job failed: ./zhammer-min.sh /test 1000 16k 1 7 The zhammer script is available at: https://github.com/numinit/nixpkgs/blob/zhammer/nixos/tests/zhammer.sh The bootstrap for it is to use a ramdisk backed file: truncate -s 4G /dev/shm/zfs zpool create -f -o ashift=12 -O canmount=off -O mountpoint=none -O xattr=sa -O dnodesize=auto -O acltype=posix -O atime=off -O relatime=on tank /dev/shm/zfs zfs create -o canmount=on -o mountpoint=/test tank/test parallel --lb --halt-on-error now,fail=1 zhammer /test 1000 16k 1 ::: $(seq $(nproc)) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-hwe-6.5 in Ubuntu. https://bugs.launchpad.net/bugs/2044657 Title: zfs block cloning file system corruption Status in linux-hwe-6.2 package in Ubuntu: Confirmed Status in linux-hwe-6.5 package in Ubuntu: Confirmed Status in zfs-linux package in Ubuntu: Confirmed Bug description: OpenZFS 2.2 reportedly has a bug where block cloning might lead to file system corruption and data loss. This was fixed in OpenZFS 2.2.1. Original bug report: https://github.com/openzfs/zfs/issues/15526 and 2.2.1 release notes: https://github.com/openzfs/zfs/releases/tag/zfs-2.2.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-hwe-6.2/+bug/2044657/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2044657] Re: zfs block cloning file system corruption
It appears upstream is prepping a 2.1.14 and 2.2.2 release that includes this fix to both branches. https://github.com/openzfs/zfs/pull/15601 https://github.com/openzfs/zfs/pull/15602 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-hwe-6.5 in Ubuntu. https://bugs.launchpad.net/bugs/2044657 Title: zfs block cloning file system corruption Status in linux-hwe-6.2 package in Ubuntu: New Status in linux-hwe-6.5 package in Ubuntu: New Status in zfs-linux package in Ubuntu: Confirmed Bug description: OpenZFS 2.2 reportedly has a bug where block cloning might lead to file system corruption and data loss. This was fixed in OpenZFS 2.2.1. Original bug report: https://github.com/openzfs/zfs/issues/15526 and 2.2.1 release notes: https://github.com/openzfs/zfs/releases/tag/zfs-2.2.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-hwe-6.2/+bug/2044657/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2044657] Re: zfs block cloning file system corruption
Apparently 2.2.1 did not fix the issue, there is an on-going PR https://github.com/openzfs/zfs/pull/15571 that has the latest fix for the data corruption. As a side note it also affects zfs 2.1 series (block cloning made it more evident, but apparently possible to happen on earlier zfs releases), would we see a linux-hwe-6.2 bump for this with a 2.1 patch? Below is the 2.1.x series PR: https://github.com/openzfs/zfs/pull/15578 ** Also affects: linux-hwe-6.2 (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to zfs-linux in Ubuntu. https://bugs.launchpad.net/bugs/2044657 Title: zfs block cloning file system corruption Status in linux-hwe-6.2 package in Ubuntu: New Status in zfs-linux package in Ubuntu: Confirmed Bug description: OpenZFS 2.2 reportedly has a bug where block cloning might lead to file system corruption and data loss. This was fixed in OpenZFS 2.2.1. Original bug report: https://github.com/openzfs/zfs/issues/15526 and 2.2.1 release notes: https://github.com/openzfs/zfs/releases/tag/zfs-2.2.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-hwe-6.2/+bug/2044657/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1942935] Re: kernel io hangs during mdcheck/resync
Comment #5 (https://bugs.launchpad.net/ubuntu/+source/linux-signed- hwe-5.11/+bug/1942935/comments/5) has been a stable workaround for me (basically revert back to a continuous resync like 18.04). My newer machines are using ZFS with raidz2 pools. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-signed-hwe-5.11 in Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync Status in linux package in Ubuntu: Confirmed Status in linux-signed-hwe-5.11 package in Ubuntu: Confirmed Bug description: It seems to always occur during an mdcheck/resync, if I am logged in via SSH it is still somewhat responsive and basic utilities like dmesg will work. But it apppears any write I/O will hang the terminal and nothing is written to syslog (presumably because it is blocked). Below is output of dmesg and cat /proc/mdstat, it appears the data check was interrupted and /proc/mdstat still shows progress, and a whole slew of hung tasks including md1_resync itself. [756484.534293] md: data-check of RAID array md0 [756484.628039] md: delaying data-check of md1 until md0 has finished (they share one or more physical units) [756493.808773] md: md0: data-check done. [756493.829760] md: data-check of RAID array md1 [778112.446410] md: md1: data-check interrupted. [810654.608102] md: data-check of RAID array md1 [832291.201064] md: md1: data-check interrupted. [899745.389485] md: data-check of RAID array md1 [921395.835305] md: md1: data-check interrupted. [921588.558834] INFO: task systemd-journal:376 blocked for more than 120 seconds. [921588.558846] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.558850] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.558854] task:systemd-journal state:D stack:0 pid: 376 ppid: 1 flags:0x0220 [921588.558859] Call Trace: [921588.558864] __schedule+0x44c/0x8a0 [921588.558872] schedule+0x4f/0xc0 [921588.558876] md_write_start+0x150/0x240 [921588.558880] ? wait_woken+0x80/0x80 [921588.558886] raid5_make_request+0x88/0x890 [raid456] [921588.558898] ? wait_woken+0x80/0x80 [921588.558901] ? mempool_kmalloc+0x17/0x20 [921588.558904] md_handle_request+0x12d/0x1a0 [921588.558907] ? __part_start_io_acct+0x51/0xf0 [921588.558912] md_submit_bio+0xca/0x100 [921588.558915] submit_bio_noacct+0x112/0x4f0 [921588.558918] ? ext4_fc_reserve_space+0x110/0x230 [921588.558922] submit_bio+0x51/0x1a0 [921588.558925] ? _cond_resched+0x19/0x30 [921588.558928] ? kmem_cache_alloc+0x38e/0x440 [921588.558932] ? ext4_init_io_end+0x1f/0x50 [921588.558936] ext4_io_submit+0x4d/0x60 [921588.558940] ext4_writepages+0x2c6/0xcd0 [921588.558944] do_writepages+0x43/0xd0 [921588.558948] ? do_writepages+0x43/0xd0 [921588.558951] ? fault_dirty_shared_page+0xa5/0x110 [921588.558955] __filemap_fdatawrite_range+0xcc/0x110 [921588.558960] file_write_and_wait_range+0x74/0xc0 [921588.558962] ext4_sync_file+0xf5/0x350 [921588.558967] vfs_fsync_range+0x49/0x80 [921588.558970] do_fsync+0x3d/0x70 [921588.558973] __x64_sys_fsync+0x14/0x20 [921588.558976] do_syscall_64+0x38/0x90 [921588.558980] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921588.558984] RIP: 0033:0x7f4c97ee832b [921588.558987] RSP: 002b:7ffdceb29e50 EFLAGS: 0293 ORIG_RAX: 004a [921588.558991] RAX: ffda RBX: 55ced34b0fa0 RCX: 7f4c97ee832b [921588.558993] RDX: 7f4c97fc8000 RSI: 55ced3487b70 RDI: 0021 [921588.558995] RBP: 0001 R08: R09: 7ffdceb29fa8 [921588.558996] R10: 7f4c97d2c848 R11: 0293 R12: 7ffdceb29fa8 [921588.558998] R13: 7ffdceb29fa0 R14: 55ced34b0fa0 R15: 55ced34bcf90 [921588.559014] INFO: task mysqld:1505 blocked for more than 120 seconds. [921588.559018] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.559022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.559025] task:mysqld state:D stack:0 pid: 1505 ppid: 1 flags:0x [921588.559030] Call Trace: [921588.559032] __schedule+0x44c/0x8a0 [921588.559036] schedule+0x4f/0xc0 [921588.559040] md_write_start+0x150/0x240 [921588.559044] ? wait_woken+0x80/0x80 [921588.559047] raid5_make_request+0x88/0x890 [raid456] [921588.559056] ? wait_woken+0x80/0x80 [921588.559059] ? mempool_kmalloc+0x17/0x20 [921588.559062] md_handle_request+0x12d/0x1a0 [921588.559065] ? __part_start_io_acct+0x51/0xf0 [921588.559068] md_submit_bio+0xca/0x100 [921588.559071] submit_bio_noacct+0x112/0x4f0 [921588.559075] submit_bio+0x51/0x1a0 [921588.559077] ? _cond_resched+0x19/0x30 [921588.559081] ? kmem_cache_alloc+0x38e/0x440 [921588.559084] ? ext4_init_io_end+0x1f/0x50 [921588.559088]
[Kernel-packages] [Bug 1942935] Re: kernel io hangs during mdcheck/resync
Yeah, that's the same issue as this one. The issue is the raid is doing a consistency check (mdcheck) and is transitioned to an "idle" state and hits a deadlock that causes all I/O through the md device to block. The workaround is to change the array state back to active. I made the changes in #5 almost a year ago and no problems, before that it pretty much hung almost every single month when the scheduled consistency check was triggered ever since upgrading to 20.04. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-signed-hwe-5.11 in Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync Status in linux package in Ubuntu: Confirmed Status in linux-signed-hwe-5.11 package in Ubuntu: Confirmed Bug description: It seems to always occur during an mdcheck/resync, if I am logged in via SSH it is still somewhat responsive and basic utilities like dmesg will work. But it apppears any write I/O will hang the terminal and nothing is written to syslog (presumably because it is blocked). Below is output of dmesg and cat /proc/mdstat, it appears the data check was interrupted and /proc/mdstat still shows progress, and a whole slew of hung tasks including md1_resync itself. [756484.534293] md: data-check of RAID array md0 [756484.628039] md: delaying data-check of md1 until md0 has finished (they share one or more physical units) [756493.808773] md: md0: data-check done. [756493.829760] md: data-check of RAID array md1 [778112.446410] md: md1: data-check interrupted. [810654.608102] md: data-check of RAID array md1 [832291.201064] md: md1: data-check interrupted. [899745.389485] md: data-check of RAID array md1 [921395.835305] md: md1: data-check interrupted. [921588.558834] INFO: task systemd-journal:376 blocked for more than 120 seconds. [921588.558846] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.558850] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.558854] task:systemd-journal state:D stack:0 pid: 376 ppid: 1 flags:0x0220 [921588.558859] Call Trace: [921588.558864] __schedule+0x44c/0x8a0 [921588.558872] schedule+0x4f/0xc0 [921588.558876] md_write_start+0x150/0x240 [921588.558880] ? wait_woken+0x80/0x80 [921588.558886] raid5_make_request+0x88/0x890 [raid456] [921588.558898] ? wait_woken+0x80/0x80 [921588.558901] ? mempool_kmalloc+0x17/0x20 [921588.558904] md_handle_request+0x12d/0x1a0 [921588.558907] ? __part_start_io_acct+0x51/0xf0 [921588.558912] md_submit_bio+0xca/0x100 [921588.558915] submit_bio_noacct+0x112/0x4f0 [921588.558918] ? ext4_fc_reserve_space+0x110/0x230 [921588.558922] submit_bio+0x51/0x1a0 [921588.558925] ? _cond_resched+0x19/0x30 [921588.558928] ? kmem_cache_alloc+0x38e/0x440 [921588.558932] ? ext4_init_io_end+0x1f/0x50 [921588.558936] ext4_io_submit+0x4d/0x60 [921588.558940] ext4_writepages+0x2c6/0xcd0 [921588.558944] do_writepages+0x43/0xd0 [921588.558948] ? do_writepages+0x43/0xd0 [921588.558951] ? fault_dirty_shared_page+0xa5/0x110 [921588.558955] __filemap_fdatawrite_range+0xcc/0x110 [921588.558960] file_write_and_wait_range+0x74/0xc0 [921588.558962] ext4_sync_file+0xf5/0x350 [921588.558967] vfs_fsync_range+0x49/0x80 [921588.558970] do_fsync+0x3d/0x70 [921588.558973] __x64_sys_fsync+0x14/0x20 [921588.558976] do_syscall_64+0x38/0x90 [921588.558980] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921588.558984] RIP: 0033:0x7f4c97ee832b [921588.558987] RSP: 002b:7ffdceb29e50 EFLAGS: 0293 ORIG_RAX: 004a [921588.558991] RAX: ffda RBX: 55ced34b0fa0 RCX: 7f4c97ee832b [921588.558993] RDX: 7f4c97fc8000 RSI: 55ced3487b70 RDI: 0021 [921588.558995] RBP: 0001 R08: R09: 7ffdceb29fa8 [921588.558996] R10: 7f4c97d2c848 R11: 0293 R12: 7ffdceb29fa8 [921588.558998] R13: 7ffdceb29fa0 R14: 55ced34b0fa0 R15: 55ced34bcf90 [921588.559014] INFO: task mysqld:1505 blocked for more than 120 seconds. [921588.559018] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.559022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.559025] task:mysqld state:D stack:0 pid: 1505 ppid: 1 flags:0x [921588.559030] Call Trace: [921588.559032] __schedule+0x44c/0x8a0 [921588.559036] schedule+0x4f/0xc0 [921588.559040] md_write_start+0x150/0x240 [921588.559044] ? wait_woken+0x80/0x80 [921588.559047] raid5_make_request+0x88/0x890 [raid456] [921588.559056] ? wait_woken+0x80/0x80 [921588.559059] ? mempool_kmalloc+0x17/0x20 [921588.559062] md_handle_request+0x12d/0x1a0 [921588.559065] ? __part_start_io_acct+0x51/0xf0 [921588.559068] md_submit_bio+0xca/0x100 [921588.559071]
[Kernel-packages] [Bug 1942935] Re: kernel io hangs during mdcheck/resync
I believe to resolve the deadlock you want to do: echo active > /sys/block/md1/md/array_state Not "idle". You should see a hung task for mdcheck in there somewhere as well, and it only occurs when the raid is resyncing (md_resync should be running), at least for me I the workaround in comment 5: https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-5.11/+bug/1942935/comments/5 Haven't had a problem since, upgraded to 22.04 since then as well with 5.15. I am pretty sure the problem is still there, the problem surfaced in 20.04 when they changed the raid consistency check. The new check pauses the check after 8 hours and triggers the deadlock, I just let it run to completion like it did in 18.04. I have not checked the patches, but it's possible you have a different problem because I don't see in either of two hung process traces any calls to md_ code. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-signed-hwe-5.11 in Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync Status in linux package in Ubuntu: Confirmed Status in linux-signed-hwe-5.11 package in Ubuntu: Confirmed Bug description: It seems to always occur during an mdcheck/resync, if I am logged in via SSH it is still somewhat responsive and basic utilities like dmesg will work. But it apppears any write I/O will hang the terminal and nothing is written to syslog (presumably because it is blocked). Below is output of dmesg and cat /proc/mdstat, it appears the data check was interrupted and /proc/mdstat still shows progress, and a whole slew of hung tasks including md1_resync itself. [756484.534293] md: data-check of RAID array md0 [756484.628039] md: delaying data-check of md1 until md0 has finished (they share one or more physical units) [756493.808773] md: md0: data-check done. [756493.829760] md: data-check of RAID array md1 [778112.446410] md: md1: data-check interrupted. [810654.608102] md: data-check of RAID array md1 [832291.201064] md: md1: data-check interrupted. [899745.389485] md: data-check of RAID array md1 [921395.835305] md: md1: data-check interrupted. [921588.558834] INFO: task systemd-journal:376 blocked for more than 120 seconds. [921588.558846] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.558850] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.558854] task:systemd-journal state:D stack:0 pid: 376 ppid: 1 flags:0x0220 [921588.558859] Call Trace: [921588.558864] __schedule+0x44c/0x8a0 [921588.558872] schedule+0x4f/0xc0 [921588.558876] md_write_start+0x150/0x240 [921588.558880] ? wait_woken+0x80/0x80 [921588.558886] raid5_make_request+0x88/0x890 [raid456] [921588.558898] ? wait_woken+0x80/0x80 [921588.558901] ? mempool_kmalloc+0x17/0x20 [921588.558904] md_handle_request+0x12d/0x1a0 [921588.558907] ? __part_start_io_acct+0x51/0xf0 [921588.558912] md_submit_bio+0xca/0x100 [921588.558915] submit_bio_noacct+0x112/0x4f0 [921588.558918] ? ext4_fc_reserve_space+0x110/0x230 [921588.558922] submit_bio+0x51/0x1a0 [921588.558925] ? _cond_resched+0x19/0x30 [921588.558928] ? kmem_cache_alloc+0x38e/0x440 [921588.558932] ? ext4_init_io_end+0x1f/0x50 [921588.558936] ext4_io_submit+0x4d/0x60 [921588.558940] ext4_writepages+0x2c6/0xcd0 [921588.558944] do_writepages+0x43/0xd0 [921588.558948] ? do_writepages+0x43/0xd0 [921588.558951] ? fault_dirty_shared_page+0xa5/0x110 [921588.558955] __filemap_fdatawrite_range+0xcc/0x110 [921588.558960] file_write_and_wait_range+0x74/0xc0 [921588.558962] ext4_sync_file+0xf5/0x350 [921588.558967] vfs_fsync_range+0x49/0x80 [921588.558970] do_fsync+0x3d/0x70 [921588.558973] __x64_sys_fsync+0x14/0x20 [921588.558976] do_syscall_64+0x38/0x90 [921588.558980] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921588.558984] RIP: 0033:0x7f4c97ee832b [921588.558987] RSP: 002b:7ffdceb29e50 EFLAGS: 0293 ORIG_RAX: 004a [921588.558991] RAX: ffda RBX: 55ced34b0fa0 RCX: 7f4c97ee832b [921588.558993] RDX: 7f4c97fc8000 RSI: 55ced3487b70 RDI: 0021 [921588.558995] RBP: 0001 R08: R09: 7ffdceb29fa8 [921588.558996] R10: 7f4c97d2c848 R11: 0293 R12: 7ffdceb29fa8 [921588.558998] R13: 7ffdceb29fa0 R14: 55ced34b0fa0 R15: 55ced34bcf90 [921588.559014] INFO: task mysqld:1505 blocked for more than 120 seconds. [921588.559018] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.559022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.559025] task:mysqld state:D stack:0 pid: 1505 ppid: 1 flags:0x [921588.559030] Call Trace: [921588.559032] __schedule+0x44c/0x8a0 [921588.559036] schedule+0x4f/0xc0
[Kernel-packages] [Bug 1942935] Re: kernel io hangs during mdcheck/resync
Looks like two patches are landing in next to resolve this: https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?h=next-20220527=8b48ec23cc51a4e7c8dbaef5f34ebe67e1a80934 https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?h=next-20220527=1e267742283a4b5a8ca65755c44166be27e9aa0f -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-signed-hwe-5.11 in Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync Status in linux package in Ubuntu: Confirmed Status in linux-signed-hwe-5.11 package in Ubuntu: Confirmed Bug description: It seems to always occur during an mdcheck/resync, if I am logged in via SSH it is still somewhat responsive and basic utilities like dmesg will work. But it apppears any write I/O will hang the terminal and nothing is written to syslog (presumably because it is blocked). Below is output of dmesg and cat /proc/mdstat, it appears the data check was interrupted and /proc/mdstat still shows progress, and a whole slew of hung tasks including md1_resync itself. [756484.534293] md: data-check of RAID array md0 [756484.628039] md: delaying data-check of md1 until md0 has finished (they share one or more physical units) [756493.808773] md: md0: data-check done. [756493.829760] md: data-check of RAID array md1 [778112.446410] md: md1: data-check interrupted. [810654.608102] md: data-check of RAID array md1 [832291.201064] md: md1: data-check interrupted. [899745.389485] md: data-check of RAID array md1 [921395.835305] md: md1: data-check interrupted. [921588.558834] INFO: task systemd-journal:376 blocked for more than 120 seconds. [921588.558846] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.558850] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.558854] task:systemd-journal state:D stack:0 pid: 376 ppid: 1 flags:0x0220 [921588.558859] Call Trace: [921588.558864] __schedule+0x44c/0x8a0 [921588.558872] schedule+0x4f/0xc0 [921588.558876] md_write_start+0x150/0x240 [921588.558880] ? wait_woken+0x80/0x80 [921588.558886] raid5_make_request+0x88/0x890 [raid456] [921588.558898] ? wait_woken+0x80/0x80 [921588.558901] ? mempool_kmalloc+0x17/0x20 [921588.558904] md_handle_request+0x12d/0x1a0 [921588.558907] ? __part_start_io_acct+0x51/0xf0 [921588.558912] md_submit_bio+0xca/0x100 [921588.558915] submit_bio_noacct+0x112/0x4f0 [921588.558918] ? ext4_fc_reserve_space+0x110/0x230 [921588.558922] submit_bio+0x51/0x1a0 [921588.558925] ? _cond_resched+0x19/0x30 [921588.558928] ? kmem_cache_alloc+0x38e/0x440 [921588.558932] ? ext4_init_io_end+0x1f/0x50 [921588.558936] ext4_io_submit+0x4d/0x60 [921588.558940] ext4_writepages+0x2c6/0xcd0 [921588.558944] do_writepages+0x43/0xd0 [921588.558948] ? do_writepages+0x43/0xd0 [921588.558951] ? fault_dirty_shared_page+0xa5/0x110 [921588.558955] __filemap_fdatawrite_range+0xcc/0x110 [921588.558960] file_write_and_wait_range+0x74/0xc0 [921588.558962] ext4_sync_file+0xf5/0x350 [921588.558967] vfs_fsync_range+0x49/0x80 [921588.558970] do_fsync+0x3d/0x70 [921588.558973] __x64_sys_fsync+0x14/0x20 [921588.558976] do_syscall_64+0x38/0x90 [921588.558980] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921588.558984] RIP: 0033:0x7f4c97ee832b [921588.558987] RSP: 002b:7ffdceb29e50 EFLAGS: 0293 ORIG_RAX: 004a [921588.558991] RAX: ffda RBX: 55ced34b0fa0 RCX: 7f4c97ee832b [921588.558993] RDX: 7f4c97fc8000 RSI: 55ced3487b70 RDI: 0021 [921588.558995] RBP: 0001 R08: R09: 7ffdceb29fa8 [921588.558996] R10: 7f4c97d2c848 R11: 0293 R12: 7ffdceb29fa8 [921588.558998] R13: 7ffdceb29fa0 R14: 55ced34b0fa0 R15: 55ced34bcf90 [921588.559014] INFO: task mysqld:1505 blocked for more than 120 seconds. [921588.559018] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.559022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.559025] task:mysqld state:D stack:0 pid: 1505 ppid: 1 flags:0x [921588.559030] Call Trace: [921588.559032] __schedule+0x44c/0x8a0 [921588.559036] schedule+0x4f/0xc0 [921588.559040] md_write_start+0x150/0x240 [921588.559044] ? wait_woken+0x80/0x80 [921588.559047] raid5_make_request+0x88/0x890 [raid456] [921588.559056] ? wait_woken+0x80/0x80 [921588.559059] ? mempool_kmalloc+0x17/0x20 [921588.559062] md_handle_request+0x12d/0x1a0 [921588.559065] ? __part_start_io_acct+0x51/0xf0 [921588.559068] md_submit_bio+0xca/0x100 [921588.559071] submit_bio_noacct+0x112/0x4f0 [921588.559075] submit_bio+0x51/0x1a0 [921588.559077] ? _cond_resched+0x19/0x30 [921588.559081] ?
[Kernel-packages] [Bug 1942935] acpidump.txt
apport information ** Attachment added: "acpidump.txt" https://bugs.launchpad.net/bugs/1942935/+attachment/5526040/+files/acpidump.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-signed-hwe-5.11 in Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync Status in linux package in Ubuntu: Confirmed Status in linux-signed-hwe-5.11 package in Ubuntu: Confirmed Bug description: It seems to always occur during an mdcheck/resync, if I am logged in via SSH it is still somewhat responsive and basic utilities like dmesg will work. But it apppears any write I/O will hang the terminal and nothing is written to syslog (presumably because it is blocked). Below is output of dmesg and cat /proc/mdstat, it appears the data check was interrupted and /proc/mdstat still shows progress, and a whole slew of hung tasks including md1_resync itself. [756484.534293] md: data-check of RAID array md0 [756484.628039] md: delaying data-check of md1 until md0 has finished (they share one or more physical units) [756493.808773] md: md0: data-check done. [756493.829760] md: data-check of RAID array md1 [778112.446410] md: md1: data-check interrupted. [810654.608102] md: data-check of RAID array md1 [832291.201064] md: md1: data-check interrupted. [899745.389485] md: data-check of RAID array md1 [921395.835305] md: md1: data-check interrupted. [921588.558834] INFO: task systemd-journal:376 blocked for more than 120 seconds. [921588.558846] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.558850] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.558854] task:systemd-journal state:D stack:0 pid: 376 ppid: 1 flags:0x0220 [921588.558859] Call Trace: [921588.558864] __schedule+0x44c/0x8a0 [921588.558872] schedule+0x4f/0xc0 [921588.558876] md_write_start+0x150/0x240 [921588.558880] ? wait_woken+0x80/0x80 [921588.558886] raid5_make_request+0x88/0x890 [raid456] [921588.558898] ? wait_woken+0x80/0x80 [921588.558901] ? mempool_kmalloc+0x17/0x20 [921588.558904] md_handle_request+0x12d/0x1a0 [921588.558907] ? __part_start_io_acct+0x51/0xf0 [921588.558912] md_submit_bio+0xca/0x100 [921588.558915] submit_bio_noacct+0x112/0x4f0 [921588.558918] ? ext4_fc_reserve_space+0x110/0x230 [921588.558922] submit_bio+0x51/0x1a0 [921588.558925] ? _cond_resched+0x19/0x30 [921588.558928] ? kmem_cache_alloc+0x38e/0x440 [921588.558932] ? ext4_init_io_end+0x1f/0x50 [921588.558936] ext4_io_submit+0x4d/0x60 [921588.558940] ext4_writepages+0x2c6/0xcd0 [921588.558944] do_writepages+0x43/0xd0 [921588.558948] ? do_writepages+0x43/0xd0 [921588.558951] ? fault_dirty_shared_page+0xa5/0x110 [921588.558955] __filemap_fdatawrite_range+0xcc/0x110 [921588.558960] file_write_and_wait_range+0x74/0xc0 [921588.558962] ext4_sync_file+0xf5/0x350 [921588.558967] vfs_fsync_range+0x49/0x80 [921588.558970] do_fsync+0x3d/0x70 [921588.558973] __x64_sys_fsync+0x14/0x20 [921588.558976] do_syscall_64+0x38/0x90 [921588.558980] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921588.558984] RIP: 0033:0x7f4c97ee832b [921588.558987] RSP: 002b:7ffdceb29e50 EFLAGS: 0293 ORIG_RAX: 004a [921588.558991] RAX: ffda RBX: 55ced34b0fa0 RCX: 7f4c97ee832b [921588.558993] RDX: 7f4c97fc8000 RSI: 55ced3487b70 RDI: 0021 [921588.558995] RBP: 0001 R08: R09: 7ffdceb29fa8 [921588.558996] R10: 7f4c97d2c848 R11: 0293 R12: 7ffdceb29fa8 [921588.558998] R13: 7ffdceb29fa0 R14: 55ced34b0fa0 R15: 55ced34bcf90 [921588.559014] INFO: task mysqld:1505 blocked for more than 120 seconds. [921588.559018] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.559022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.559025] task:mysqld state:D stack:0 pid: 1505 ppid: 1 flags:0x [921588.559030] Call Trace: [921588.559032] __schedule+0x44c/0x8a0 [921588.559036] schedule+0x4f/0xc0 [921588.559040] md_write_start+0x150/0x240 [921588.559044] ? wait_woken+0x80/0x80 [921588.559047] raid5_make_request+0x88/0x890 [raid456] [921588.559056] ? wait_woken+0x80/0x80 [921588.559059] ? mempool_kmalloc+0x17/0x20 [921588.559062] md_handle_request+0x12d/0x1a0 [921588.559065] ? __part_start_io_acct+0x51/0xf0 [921588.559068] md_submit_bio+0xca/0x100 [921588.559071] submit_bio_noacct+0x112/0x4f0 [921588.559075] submit_bio+0x51/0x1a0 [921588.559077] ? _cond_resched+0x19/0x30 [921588.559081] ? kmem_cache_alloc+0x38e/0x440 [921588.559084] ? ext4_init_io_end+0x1f/0x50 [921588.559088] ext4_io_submit+0x4d/0x60 [921588.559091] ext4_writepages+0x2c6/0xcd0 [921588.559094] ?
[Kernel-packages] [Bug 1942935] Re: kernel io hangs during mdcheck/resync
Same issue on impish 5.13.13 kernel, running in VBox. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-signed-hwe-5.11 in Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync Status in linux package in Ubuntu: Confirmed Status in linux-signed-hwe-5.11 package in Ubuntu: Confirmed Bug description: It seems to always occur during an mdcheck/resync, if I am logged in via SSH it is still somewhat responsive and basic utilities like dmesg will work. But it apppears any write I/O will hang the terminal and nothing is written to syslog (presumably because it is blocked). Below is output of dmesg and cat /proc/mdstat, it appears the data check was interrupted and /proc/mdstat still shows progress, and a whole slew of hung tasks including md1_resync itself. [756484.534293] md: data-check of RAID array md0 [756484.628039] md: delaying data-check of md1 until md0 has finished (they share one or more physical units) [756493.808773] md: md0: data-check done. [756493.829760] md: data-check of RAID array md1 [778112.446410] md: md1: data-check interrupted. [810654.608102] md: data-check of RAID array md1 [832291.201064] md: md1: data-check interrupted. [899745.389485] md: data-check of RAID array md1 [921395.835305] md: md1: data-check interrupted. [921588.558834] INFO: task systemd-journal:376 blocked for more than 120 seconds. [921588.558846] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.558850] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.558854] task:systemd-journal state:D stack:0 pid: 376 ppid: 1 flags:0x0220 [921588.558859] Call Trace: [921588.558864] __schedule+0x44c/0x8a0 [921588.558872] schedule+0x4f/0xc0 [921588.558876] md_write_start+0x150/0x240 [921588.558880] ? wait_woken+0x80/0x80 [921588.558886] raid5_make_request+0x88/0x890 [raid456] [921588.558898] ? wait_woken+0x80/0x80 [921588.558901] ? mempool_kmalloc+0x17/0x20 [921588.558904] md_handle_request+0x12d/0x1a0 [921588.558907] ? __part_start_io_acct+0x51/0xf0 [921588.558912] md_submit_bio+0xca/0x100 [921588.558915] submit_bio_noacct+0x112/0x4f0 [921588.558918] ? ext4_fc_reserve_space+0x110/0x230 [921588.558922] submit_bio+0x51/0x1a0 [921588.558925] ? _cond_resched+0x19/0x30 [921588.558928] ? kmem_cache_alloc+0x38e/0x440 [921588.558932] ? ext4_init_io_end+0x1f/0x50 [921588.558936] ext4_io_submit+0x4d/0x60 [921588.558940] ext4_writepages+0x2c6/0xcd0 [921588.558944] do_writepages+0x43/0xd0 [921588.558948] ? do_writepages+0x43/0xd0 [921588.558951] ? fault_dirty_shared_page+0xa5/0x110 [921588.558955] __filemap_fdatawrite_range+0xcc/0x110 [921588.558960] file_write_and_wait_range+0x74/0xc0 [921588.558962] ext4_sync_file+0xf5/0x350 [921588.558967] vfs_fsync_range+0x49/0x80 [921588.558970] do_fsync+0x3d/0x70 [921588.558973] __x64_sys_fsync+0x14/0x20 [921588.558976] do_syscall_64+0x38/0x90 [921588.558980] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921588.558984] RIP: 0033:0x7f4c97ee832b [921588.558987] RSP: 002b:7ffdceb29e50 EFLAGS: 0293 ORIG_RAX: 004a [921588.558991] RAX: ffda RBX: 55ced34b0fa0 RCX: 7f4c97ee832b [921588.558993] RDX: 7f4c97fc8000 RSI: 55ced3487b70 RDI: 0021 [921588.558995] RBP: 0001 R08: R09: 7ffdceb29fa8 [921588.558996] R10: 7f4c97d2c848 R11: 0293 R12: 7ffdceb29fa8 [921588.558998] R13: 7ffdceb29fa0 R14: 55ced34b0fa0 R15: 55ced34bcf90 [921588.559014] INFO: task mysqld:1505 blocked for more than 120 seconds. [921588.559018] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.559022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.559025] task:mysqld state:D stack:0 pid: 1505 ppid: 1 flags:0x [921588.559030] Call Trace: [921588.559032] __schedule+0x44c/0x8a0 [921588.559036] schedule+0x4f/0xc0 [921588.559040] md_write_start+0x150/0x240 [921588.559044] ? wait_woken+0x80/0x80 [921588.559047] raid5_make_request+0x88/0x890 [raid456] [921588.559056] ? wait_woken+0x80/0x80 [921588.559059] ? mempool_kmalloc+0x17/0x20 [921588.559062] md_handle_request+0x12d/0x1a0 [921588.559065] ? __part_start_io_acct+0x51/0xf0 [921588.559068] md_submit_bio+0xca/0x100 [921588.559071] submit_bio_noacct+0x112/0x4f0 [921588.559075] submit_bio+0x51/0x1a0 [921588.559077] ? _cond_resched+0x19/0x30 [921588.559081] ? kmem_cache_alloc+0x38e/0x440 [921588.559084] ? ext4_init_io_end+0x1f/0x50 [921588.559088] ext4_io_submit+0x4d/0x60 [921588.559091] ext4_writepages+0x2c6/0xcd0 [921588.559094] ? __schedule+0x454/0x8a0 [921588.559097] ? hrtimer_start_range_ns+0x1aa/0x2f0 [921588.559100] ?
[Kernel-packages] [Bug 1942935] UdevDb.txt
apport information ** Attachment added: "UdevDb.txt" https://bugs.launchpad.net/bugs/1942935/+attachment/5526038/+files/UdevDb.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-signed-hwe-5.11 in Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync Status in linux package in Ubuntu: Confirmed Status in linux-signed-hwe-5.11 package in Ubuntu: Confirmed Bug description: It seems to always occur during an mdcheck/resync, if I am logged in via SSH it is still somewhat responsive and basic utilities like dmesg will work. But it apppears any write I/O will hang the terminal and nothing is written to syslog (presumably because it is blocked). Below is output of dmesg and cat /proc/mdstat, it appears the data check was interrupted and /proc/mdstat still shows progress, and a whole slew of hung tasks including md1_resync itself. [756484.534293] md: data-check of RAID array md0 [756484.628039] md: delaying data-check of md1 until md0 has finished (they share one or more physical units) [756493.808773] md: md0: data-check done. [756493.829760] md: data-check of RAID array md1 [778112.446410] md: md1: data-check interrupted. [810654.608102] md: data-check of RAID array md1 [832291.201064] md: md1: data-check interrupted. [899745.389485] md: data-check of RAID array md1 [921395.835305] md: md1: data-check interrupted. [921588.558834] INFO: task systemd-journal:376 blocked for more than 120 seconds. [921588.558846] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.558850] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.558854] task:systemd-journal state:D stack:0 pid: 376 ppid: 1 flags:0x0220 [921588.558859] Call Trace: [921588.558864] __schedule+0x44c/0x8a0 [921588.558872] schedule+0x4f/0xc0 [921588.558876] md_write_start+0x150/0x240 [921588.558880] ? wait_woken+0x80/0x80 [921588.558886] raid5_make_request+0x88/0x890 [raid456] [921588.558898] ? wait_woken+0x80/0x80 [921588.558901] ? mempool_kmalloc+0x17/0x20 [921588.558904] md_handle_request+0x12d/0x1a0 [921588.558907] ? __part_start_io_acct+0x51/0xf0 [921588.558912] md_submit_bio+0xca/0x100 [921588.558915] submit_bio_noacct+0x112/0x4f0 [921588.558918] ? ext4_fc_reserve_space+0x110/0x230 [921588.558922] submit_bio+0x51/0x1a0 [921588.558925] ? _cond_resched+0x19/0x30 [921588.558928] ? kmem_cache_alloc+0x38e/0x440 [921588.558932] ? ext4_init_io_end+0x1f/0x50 [921588.558936] ext4_io_submit+0x4d/0x60 [921588.558940] ext4_writepages+0x2c6/0xcd0 [921588.558944] do_writepages+0x43/0xd0 [921588.558948] ? do_writepages+0x43/0xd0 [921588.558951] ? fault_dirty_shared_page+0xa5/0x110 [921588.558955] __filemap_fdatawrite_range+0xcc/0x110 [921588.558960] file_write_and_wait_range+0x74/0xc0 [921588.558962] ext4_sync_file+0xf5/0x350 [921588.558967] vfs_fsync_range+0x49/0x80 [921588.558970] do_fsync+0x3d/0x70 [921588.558973] __x64_sys_fsync+0x14/0x20 [921588.558976] do_syscall_64+0x38/0x90 [921588.558980] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921588.558984] RIP: 0033:0x7f4c97ee832b [921588.558987] RSP: 002b:7ffdceb29e50 EFLAGS: 0293 ORIG_RAX: 004a [921588.558991] RAX: ffda RBX: 55ced34b0fa0 RCX: 7f4c97ee832b [921588.558993] RDX: 7f4c97fc8000 RSI: 55ced3487b70 RDI: 0021 [921588.558995] RBP: 0001 R08: R09: 7ffdceb29fa8 [921588.558996] R10: 7f4c97d2c848 R11: 0293 R12: 7ffdceb29fa8 [921588.558998] R13: 7ffdceb29fa0 R14: 55ced34b0fa0 R15: 55ced34bcf90 [921588.559014] INFO: task mysqld:1505 blocked for more than 120 seconds. [921588.559018] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.559022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.559025] task:mysqld state:D stack:0 pid: 1505 ppid: 1 flags:0x [921588.559030] Call Trace: [921588.559032] __schedule+0x44c/0x8a0 [921588.559036] schedule+0x4f/0xc0 [921588.559040] md_write_start+0x150/0x240 [921588.559044] ? wait_woken+0x80/0x80 [921588.559047] raid5_make_request+0x88/0x890 [raid456] [921588.559056] ? wait_woken+0x80/0x80 [921588.559059] ? mempool_kmalloc+0x17/0x20 [921588.559062] md_handle_request+0x12d/0x1a0 [921588.559065] ? __part_start_io_acct+0x51/0xf0 [921588.559068] md_submit_bio+0xca/0x100 [921588.559071] submit_bio_noacct+0x112/0x4f0 [921588.559075] submit_bio+0x51/0x1a0 [921588.559077] ? _cond_resched+0x19/0x30 [921588.559081] ? kmem_cache_alloc+0x38e/0x440 [921588.559084] ? ext4_init_io_end+0x1f/0x50 [921588.559088] ext4_io_submit+0x4d/0x60 [921588.559091] ext4_writepages+0x2c6/0xcd0 [921588.559094] ? __schedule+0x454/0x8a0
[Kernel-packages] [Bug 1942935] ProcInterrupts.txt
apport information ** Attachment added: "ProcInterrupts.txt" https://bugs.launchpad.net/bugs/1942935/+attachment/5526036/+files/ProcInterrupts.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-signed-hwe-5.11 in Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync Status in linux package in Ubuntu: Confirmed Status in linux-signed-hwe-5.11 package in Ubuntu: Confirmed Bug description: It seems to always occur during an mdcheck/resync, if I am logged in via SSH it is still somewhat responsive and basic utilities like dmesg will work. But it apppears any write I/O will hang the terminal and nothing is written to syslog (presumably because it is blocked). Below is output of dmesg and cat /proc/mdstat, it appears the data check was interrupted and /proc/mdstat still shows progress, and a whole slew of hung tasks including md1_resync itself. [756484.534293] md: data-check of RAID array md0 [756484.628039] md: delaying data-check of md1 until md0 has finished (they share one or more physical units) [756493.808773] md: md0: data-check done. [756493.829760] md: data-check of RAID array md1 [778112.446410] md: md1: data-check interrupted. [810654.608102] md: data-check of RAID array md1 [832291.201064] md: md1: data-check interrupted. [899745.389485] md: data-check of RAID array md1 [921395.835305] md: md1: data-check interrupted. [921588.558834] INFO: task systemd-journal:376 blocked for more than 120 seconds. [921588.558846] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.558850] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.558854] task:systemd-journal state:D stack:0 pid: 376 ppid: 1 flags:0x0220 [921588.558859] Call Trace: [921588.558864] __schedule+0x44c/0x8a0 [921588.558872] schedule+0x4f/0xc0 [921588.558876] md_write_start+0x150/0x240 [921588.558880] ? wait_woken+0x80/0x80 [921588.558886] raid5_make_request+0x88/0x890 [raid456] [921588.558898] ? wait_woken+0x80/0x80 [921588.558901] ? mempool_kmalloc+0x17/0x20 [921588.558904] md_handle_request+0x12d/0x1a0 [921588.558907] ? __part_start_io_acct+0x51/0xf0 [921588.558912] md_submit_bio+0xca/0x100 [921588.558915] submit_bio_noacct+0x112/0x4f0 [921588.558918] ? ext4_fc_reserve_space+0x110/0x230 [921588.558922] submit_bio+0x51/0x1a0 [921588.558925] ? _cond_resched+0x19/0x30 [921588.558928] ? kmem_cache_alloc+0x38e/0x440 [921588.558932] ? ext4_init_io_end+0x1f/0x50 [921588.558936] ext4_io_submit+0x4d/0x60 [921588.558940] ext4_writepages+0x2c6/0xcd0 [921588.558944] do_writepages+0x43/0xd0 [921588.558948] ? do_writepages+0x43/0xd0 [921588.558951] ? fault_dirty_shared_page+0xa5/0x110 [921588.558955] __filemap_fdatawrite_range+0xcc/0x110 [921588.558960] file_write_and_wait_range+0x74/0xc0 [921588.558962] ext4_sync_file+0xf5/0x350 [921588.558967] vfs_fsync_range+0x49/0x80 [921588.558970] do_fsync+0x3d/0x70 [921588.558973] __x64_sys_fsync+0x14/0x20 [921588.558976] do_syscall_64+0x38/0x90 [921588.558980] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921588.558984] RIP: 0033:0x7f4c97ee832b [921588.558987] RSP: 002b:7ffdceb29e50 EFLAGS: 0293 ORIG_RAX: 004a [921588.558991] RAX: ffda RBX: 55ced34b0fa0 RCX: 7f4c97ee832b [921588.558993] RDX: 7f4c97fc8000 RSI: 55ced3487b70 RDI: 0021 [921588.558995] RBP: 0001 R08: R09: 7ffdceb29fa8 [921588.558996] R10: 7f4c97d2c848 R11: 0293 R12: 7ffdceb29fa8 [921588.558998] R13: 7ffdceb29fa0 R14: 55ced34b0fa0 R15: 55ced34bcf90 [921588.559014] INFO: task mysqld:1505 blocked for more than 120 seconds. [921588.559018] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.559022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.559025] task:mysqld state:D stack:0 pid: 1505 ppid: 1 flags:0x [921588.559030] Call Trace: [921588.559032] __schedule+0x44c/0x8a0 [921588.559036] schedule+0x4f/0xc0 [921588.559040] md_write_start+0x150/0x240 [921588.559044] ? wait_woken+0x80/0x80 [921588.559047] raid5_make_request+0x88/0x890 [raid456] [921588.559056] ? wait_woken+0x80/0x80 [921588.559059] ? mempool_kmalloc+0x17/0x20 [921588.559062] md_handle_request+0x12d/0x1a0 [921588.559065] ? __part_start_io_acct+0x51/0xf0 [921588.559068] md_submit_bio+0xca/0x100 [921588.559071] submit_bio_noacct+0x112/0x4f0 [921588.559075] submit_bio+0x51/0x1a0 [921588.559077] ? _cond_resched+0x19/0x30 [921588.559081] ? kmem_cache_alloc+0x38e/0x440 [921588.559084] ? ext4_init_io_end+0x1f/0x50 [921588.559088] ext4_io_submit+0x4d/0x60 [921588.559091] ext4_writepages+0x2c6/0xcd0 [921588.559094] ?
[Kernel-packages] [Bug 1942935] ProcModules.txt
apport information ** Attachment added: "ProcModules.txt" https://bugs.launchpad.net/bugs/1942935/+attachment/5526037/+files/ProcModules.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-signed-hwe-5.11 in Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync Status in linux package in Ubuntu: Confirmed Status in linux-signed-hwe-5.11 package in Ubuntu: Confirmed Bug description: It seems to always occur during an mdcheck/resync, if I am logged in via SSH it is still somewhat responsive and basic utilities like dmesg will work. But it apppears any write I/O will hang the terminal and nothing is written to syslog (presumably because it is blocked). Below is output of dmesg and cat /proc/mdstat, it appears the data check was interrupted and /proc/mdstat still shows progress, and a whole slew of hung tasks including md1_resync itself. [756484.534293] md: data-check of RAID array md0 [756484.628039] md: delaying data-check of md1 until md0 has finished (they share one or more physical units) [756493.808773] md: md0: data-check done. [756493.829760] md: data-check of RAID array md1 [778112.446410] md: md1: data-check interrupted. [810654.608102] md: data-check of RAID array md1 [832291.201064] md: md1: data-check interrupted. [899745.389485] md: data-check of RAID array md1 [921395.835305] md: md1: data-check interrupted. [921588.558834] INFO: task systemd-journal:376 blocked for more than 120 seconds. [921588.558846] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.558850] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.558854] task:systemd-journal state:D stack:0 pid: 376 ppid: 1 flags:0x0220 [921588.558859] Call Trace: [921588.558864] __schedule+0x44c/0x8a0 [921588.558872] schedule+0x4f/0xc0 [921588.558876] md_write_start+0x150/0x240 [921588.558880] ? wait_woken+0x80/0x80 [921588.558886] raid5_make_request+0x88/0x890 [raid456] [921588.558898] ? wait_woken+0x80/0x80 [921588.558901] ? mempool_kmalloc+0x17/0x20 [921588.558904] md_handle_request+0x12d/0x1a0 [921588.558907] ? __part_start_io_acct+0x51/0xf0 [921588.558912] md_submit_bio+0xca/0x100 [921588.558915] submit_bio_noacct+0x112/0x4f0 [921588.558918] ? ext4_fc_reserve_space+0x110/0x230 [921588.558922] submit_bio+0x51/0x1a0 [921588.558925] ? _cond_resched+0x19/0x30 [921588.558928] ? kmem_cache_alloc+0x38e/0x440 [921588.558932] ? ext4_init_io_end+0x1f/0x50 [921588.558936] ext4_io_submit+0x4d/0x60 [921588.558940] ext4_writepages+0x2c6/0xcd0 [921588.558944] do_writepages+0x43/0xd0 [921588.558948] ? do_writepages+0x43/0xd0 [921588.558951] ? fault_dirty_shared_page+0xa5/0x110 [921588.558955] __filemap_fdatawrite_range+0xcc/0x110 [921588.558960] file_write_and_wait_range+0x74/0xc0 [921588.558962] ext4_sync_file+0xf5/0x350 [921588.558967] vfs_fsync_range+0x49/0x80 [921588.558970] do_fsync+0x3d/0x70 [921588.558973] __x64_sys_fsync+0x14/0x20 [921588.558976] do_syscall_64+0x38/0x90 [921588.558980] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921588.558984] RIP: 0033:0x7f4c97ee832b [921588.558987] RSP: 002b:7ffdceb29e50 EFLAGS: 0293 ORIG_RAX: 004a [921588.558991] RAX: ffda RBX: 55ced34b0fa0 RCX: 7f4c97ee832b [921588.558993] RDX: 7f4c97fc8000 RSI: 55ced3487b70 RDI: 0021 [921588.558995] RBP: 0001 R08: R09: 7ffdceb29fa8 [921588.558996] R10: 7f4c97d2c848 R11: 0293 R12: 7ffdceb29fa8 [921588.558998] R13: 7ffdceb29fa0 R14: 55ced34b0fa0 R15: 55ced34bcf90 [921588.559014] INFO: task mysqld:1505 blocked for more than 120 seconds. [921588.559018] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.559022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.559025] task:mysqld state:D stack:0 pid: 1505 ppid: 1 flags:0x [921588.559030] Call Trace: [921588.559032] __schedule+0x44c/0x8a0 [921588.559036] schedule+0x4f/0xc0 [921588.559040] md_write_start+0x150/0x240 [921588.559044] ? wait_woken+0x80/0x80 [921588.559047] raid5_make_request+0x88/0x890 [raid456] [921588.559056] ? wait_woken+0x80/0x80 [921588.559059] ? mempool_kmalloc+0x17/0x20 [921588.559062] md_handle_request+0x12d/0x1a0 [921588.559065] ? __part_start_io_acct+0x51/0xf0 [921588.559068] md_submit_bio+0xca/0x100 [921588.559071] submit_bio_noacct+0x112/0x4f0 [921588.559075] submit_bio+0x51/0x1a0 [921588.559077] ? _cond_resched+0x19/0x30 [921588.559081] ? kmem_cache_alloc+0x38e/0x440 [921588.559084] ? ext4_init_io_end+0x1f/0x50 [921588.559088] ext4_io_submit+0x4d/0x60 [921588.559091] ext4_writepages+0x2c6/0xcd0 [921588.559094] ?
[Kernel-packages] [Bug 1942935] WifiSyslog.txt
apport information ** Attachment added: "WifiSyslog.txt" https://bugs.launchpad.net/bugs/1942935/+attachment/5526039/+files/WifiSyslog.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-signed-hwe-5.11 in Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync Status in linux package in Ubuntu: Confirmed Status in linux-signed-hwe-5.11 package in Ubuntu: Confirmed Bug description: It seems to always occur during an mdcheck/resync, if I am logged in via SSH it is still somewhat responsive and basic utilities like dmesg will work. But it apppears any write I/O will hang the terminal and nothing is written to syslog (presumably because it is blocked). Below is output of dmesg and cat /proc/mdstat, it appears the data check was interrupted and /proc/mdstat still shows progress, and a whole slew of hung tasks including md1_resync itself. [756484.534293] md: data-check of RAID array md0 [756484.628039] md: delaying data-check of md1 until md0 has finished (they share one or more physical units) [756493.808773] md: md0: data-check done. [756493.829760] md: data-check of RAID array md1 [778112.446410] md: md1: data-check interrupted. [810654.608102] md: data-check of RAID array md1 [832291.201064] md: md1: data-check interrupted. [899745.389485] md: data-check of RAID array md1 [921395.835305] md: md1: data-check interrupted. [921588.558834] INFO: task systemd-journal:376 blocked for more than 120 seconds. [921588.558846] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.558850] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.558854] task:systemd-journal state:D stack:0 pid: 376 ppid: 1 flags:0x0220 [921588.558859] Call Trace: [921588.558864] __schedule+0x44c/0x8a0 [921588.558872] schedule+0x4f/0xc0 [921588.558876] md_write_start+0x150/0x240 [921588.558880] ? wait_woken+0x80/0x80 [921588.558886] raid5_make_request+0x88/0x890 [raid456] [921588.558898] ? wait_woken+0x80/0x80 [921588.558901] ? mempool_kmalloc+0x17/0x20 [921588.558904] md_handle_request+0x12d/0x1a0 [921588.558907] ? __part_start_io_acct+0x51/0xf0 [921588.558912] md_submit_bio+0xca/0x100 [921588.558915] submit_bio_noacct+0x112/0x4f0 [921588.558918] ? ext4_fc_reserve_space+0x110/0x230 [921588.558922] submit_bio+0x51/0x1a0 [921588.558925] ? _cond_resched+0x19/0x30 [921588.558928] ? kmem_cache_alloc+0x38e/0x440 [921588.558932] ? ext4_init_io_end+0x1f/0x50 [921588.558936] ext4_io_submit+0x4d/0x60 [921588.558940] ext4_writepages+0x2c6/0xcd0 [921588.558944] do_writepages+0x43/0xd0 [921588.558948] ? do_writepages+0x43/0xd0 [921588.558951] ? fault_dirty_shared_page+0xa5/0x110 [921588.558955] __filemap_fdatawrite_range+0xcc/0x110 [921588.558960] file_write_and_wait_range+0x74/0xc0 [921588.558962] ext4_sync_file+0xf5/0x350 [921588.558967] vfs_fsync_range+0x49/0x80 [921588.558970] do_fsync+0x3d/0x70 [921588.558973] __x64_sys_fsync+0x14/0x20 [921588.558976] do_syscall_64+0x38/0x90 [921588.558980] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921588.558984] RIP: 0033:0x7f4c97ee832b [921588.558987] RSP: 002b:7ffdceb29e50 EFLAGS: 0293 ORIG_RAX: 004a [921588.558991] RAX: ffda RBX: 55ced34b0fa0 RCX: 7f4c97ee832b [921588.558993] RDX: 7f4c97fc8000 RSI: 55ced3487b70 RDI: 0021 [921588.558995] RBP: 0001 R08: R09: 7ffdceb29fa8 [921588.558996] R10: 7f4c97d2c848 R11: 0293 R12: 7ffdceb29fa8 [921588.558998] R13: 7ffdceb29fa0 R14: 55ced34b0fa0 R15: 55ced34bcf90 [921588.559014] INFO: task mysqld:1505 blocked for more than 120 seconds. [921588.559018] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.559022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.559025] task:mysqld state:D stack:0 pid: 1505 ppid: 1 flags:0x [921588.559030] Call Trace: [921588.559032] __schedule+0x44c/0x8a0 [921588.559036] schedule+0x4f/0xc0 [921588.559040] md_write_start+0x150/0x240 [921588.559044] ? wait_woken+0x80/0x80 [921588.559047] raid5_make_request+0x88/0x890 [raid456] [921588.559056] ? wait_woken+0x80/0x80 [921588.559059] ? mempool_kmalloc+0x17/0x20 [921588.559062] md_handle_request+0x12d/0x1a0 [921588.559065] ? __part_start_io_acct+0x51/0xf0 [921588.559068] md_submit_bio+0xca/0x100 [921588.559071] submit_bio_noacct+0x112/0x4f0 [921588.559075] submit_bio+0x51/0x1a0 [921588.559077] ? _cond_resched+0x19/0x30 [921588.559081] ? kmem_cache_alloc+0x38e/0x440 [921588.559084] ? ext4_init_io_end+0x1f/0x50 [921588.559088] ext4_io_submit+0x4d/0x60 [921588.559091] ext4_writepages+0x2c6/0xcd0 [921588.559094] ?
[Kernel-packages] [Bug 1942935] ProcCpuinfo.txt
apport information ** Attachment added: "ProcCpuinfo.txt" https://bugs.launchpad.net/bugs/1942935/+attachment/5526034/+files/ProcCpuinfo.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-signed-hwe-5.11 in Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync Status in linux package in Ubuntu: Confirmed Status in linux-signed-hwe-5.11 package in Ubuntu: Confirmed Bug description: It seems to always occur during an mdcheck/resync, if I am logged in via SSH it is still somewhat responsive and basic utilities like dmesg will work. But it apppears any write I/O will hang the terminal and nothing is written to syslog (presumably because it is blocked). Below is output of dmesg and cat /proc/mdstat, it appears the data check was interrupted and /proc/mdstat still shows progress, and a whole slew of hung tasks including md1_resync itself. [756484.534293] md: data-check of RAID array md0 [756484.628039] md: delaying data-check of md1 until md0 has finished (they share one or more physical units) [756493.808773] md: md0: data-check done. [756493.829760] md: data-check of RAID array md1 [778112.446410] md: md1: data-check interrupted. [810654.608102] md: data-check of RAID array md1 [832291.201064] md: md1: data-check interrupted. [899745.389485] md: data-check of RAID array md1 [921395.835305] md: md1: data-check interrupted. [921588.558834] INFO: task systemd-journal:376 blocked for more than 120 seconds. [921588.558846] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.558850] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.558854] task:systemd-journal state:D stack:0 pid: 376 ppid: 1 flags:0x0220 [921588.558859] Call Trace: [921588.558864] __schedule+0x44c/0x8a0 [921588.558872] schedule+0x4f/0xc0 [921588.558876] md_write_start+0x150/0x240 [921588.558880] ? wait_woken+0x80/0x80 [921588.558886] raid5_make_request+0x88/0x890 [raid456] [921588.558898] ? wait_woken+0x80/0x80 [921588.558901] ? mempool_kmalloc+0x17/0x20 [921588.558904] md_handle_request+0x12d/0x1a0 [921588.558907] ? __part_start_io_acct+0x51/0xf0 [921588.558912] md_submit_bio+0xca/0x100 [921588.558915] submit_bio_noacct+0x112/0x4f0 [921588.558918] ? ext4_fc_reserve_space+0x110/0x230 [921588.558922] submit_bio+0x51/0x1a0 [921588.558925] ? _cond_resched+0x19/0x30 [921588.558928] ? kmem_cache_alloc+0x38e/0x440 [921588.558932] ? ext4_init_io_end+0x1f/0x50 [921588.558936] ext4_io_submit+0x4d/0x60 [921588.558940] ext4_writepages+0x2c6/0xcd0 [921588.558944] do_writepages+0x43/0xd0 [921588.558948] ? do_writepages+0x43/0xd0 [921588.558951] ? fault_dirty_shared_page+0xa5/0x110 [921588.558955] __filemap_fdatawrite_range+0xcc/0x110 [921588.558960] file_write_and_wait_range+0x74/0xc0 [921588.558962] ext4_sync_file+0xf5/0x350 [921588.558967] vfs_fsync_range+0x49/0x80 [921588.558970] do_fsync+0x3d/0x70 [921588.558973] __x64_sys_fsync+0x14/0x20 [921588.558976] do_syscall_64+0x38/0x90 [921588.558980] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921588.558984] RIP: 0033:0x7f4c97ee832b [921588.558987] RSP: 002b:7ffdceb29e50 EFLAGS: 0293 ORIG_RAX: 004a [921588.558991] RAX: ffda RBX: 55ced34b0fa0 RCX: 7f4c97ee832b [921588.558993] RDX: 7f4c97fc8000 RSI: 55ced3487b70 RDI: 0021 [921588.558995] RBP: 0001 R08: R09: 7ffdceb29fa8 [921588.558996] R10: 7f4c97d2c848 R11: 0293 R12: 7ffdceb29fa8 [921588.558998] R13: 7ffdceb29fa0 R14: 55ced34b0fa0 R15: 55ced34bcf90 [921588.559014] INFO: task mysqld:1505 blocked for more than 120 seconds. [921588.559018] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.559022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.559025] task:mysqld state:D stack:0 pid: 1505 ppid: 1 flags:0x [921588.559030] Call Trace: [921588.559032] __schedule+0x44c/0x8a0 [921588.559036] schedule+0x4f/0xc0 [921588.559040] md_write_start+0x150/0x240 [921588.559044] ? wait_woken+0x80/0x80 [921588.559047] raid5_make_request+0x88/0x890 [raid456] [921588.559056] ? wait_woken+0x80/0x80 [921588.559059] ? mempool_kmalloc+0x17/0x20 [921588.559062] md_handle_request+0x12d/0x1a0 [921588.559065] ? __part_start_io_acct+0x51/0xf0 [921588.559068] md_submit_bio+0xca/0x100 [921588.559071] submit_bio_noacct+0x112/0x4f0 [921588.559075] submit_bio+0x51/0x1a0 [921588.559077] ? _cond_resched+0x19/0x30 [921588.559081] ? kmem_cache_alloc+0x38e/0x440 [921588.559084] ? ext4_init_io_end+0x1f/0x50 [921588.559088] ext4_io_submit+0x4d/0x60 [921588.559091] ext4_writepages+0x2c6/0xcd0 [921588.559094] ?
[Kernel-packages] [Bug 1942935] ProcCpuinfoMinimal.txt
apport information ** Attachment added: "ProcCpuinfoMinimal.txt" https://bugs.launchpad.net/bugs/1942935/+attachment/5526035/+files/ProcCpuinfoMinimal.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-signed-hwe-5.11 in Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync Status in linux package in Ubuntu: Confirmed Status in linux-signed-hwe-5.11 package in Ubuntu: Confirmed Bug description: It seems to always occur during an mdcheck/resync, if I am logged in via SSH it is still somewhat responsive and basic utilities like dmesg will work. But it apppears any write I/O will hang the terminal and nothing is written to syslog (presumably because it is blocked). Below is output of dmesg and cat /proc/mdstat, it appears the data check was interrupted and /proc/mdstat still shows progress, and a whole slew of hung tasks including md1_resync itself. [756484.534293] md: data-check of RAID array md0 [756484.628039] md: delaying data-check of md1 until md0 has finished (they share one or more physical units) [756493.808773] md: md0: data-check done. [756493.829760] md: data-check of RAID array md1 [778112.446410] md: md1: data-check interrupted. [810654.608102] md: data-check of RAID array md1 [832291.201064] md: md1: data-check interrupted. [899745.389485] md: data-check of RAID array md1 [921395.835305] md: md1: data-check interrupted. [921588.558834] INFO: task systemd-journal:376 blocked for more than 120 seconds. [921588.558846] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.558850] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.558854] task:systemd-journal state:D stack:0 pid: 376 ppid: 1 flags:0x0220 [921588.558859] Call Trace: [921588.558864] __schedule+0x44c/0x8a0 [921588.558872] schedule+0x4f/0xc0 [921588.558876] md_write_start+0x150/0x240 [921588.558880] ? wait_woken+0x80/0x80 [921588.558886] raid5_make_request+0x88/0x890 [raid456] [921588.558898] ? wait_woken+0x80/0x80 [921588.558901] ? mempool_kmalloc+0x17/0x20 [921588.558904] md_handle_request+0x12d/0x1a0 [921588.558907] ? __part_start_io_acct+0x51/0xf0 [921588.558912] md_submit_bio+0xca/0x100 [921588.558915] submit_bio_noacct+0x112/0x4f0 [921588.558918] ? ext4_fc_reserve_space+0x110/0x230 [921588.558922] submit_bio+0x51/0x1a0 [921588.558925] ? _cond_resched+0x19/0x30 [921588.558928] ? kmem_cache_alloc+0x38e/0x440 [921588.558932] ? ext4_init_io_end+0x1f/0x50 [921588.558936] ext4_io_submit+0x4d/0x60 [921588.558940] ext4_writepages+0x2c6/0xcd0 [921588.558944] do_writepages+0x43/0xd0 [921588.558948] ? do_writepages+0x43/0xd0 [921588.558951] ? fault_dirty_shared_page+0xa5/0x110 [921588.558955] __filemap_fdatawrite_range+0xcc/0x110 [921588.558960] file_write_and_wait_range+0x74/0xc0 [921588.558962] ext4_sync_file+0xf5/0x350 [921588.558967] vfs_fsync_range+0x49/0x80 [921588.558970] do_fsync+0x3d/0x70 [921588.558973] __x64_sys_fsync+0x14/0x20 [921588.558976] do_syscall_64+0x38/0x90 [921588.558980] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921588.558984] RIP: 0033:0x7f4c97ee832b [921588.558987] RSP: 002b:7ffdceb29e50 EFLAGS: 0293 ORIG_RAX: 004a [921588.558991] RAX: ffda RBX: 55ced34b0fa0 RCX: 7f4c97ee832b [921588.558993] RDX: 7f4c97fc8000 RSI: 55ced3487b70 RDI: 0021 [921588.558995] RBP: 0001 R08: R09: 7ffdceb29fa8 [921588.558996] R10: 7f4c97d2c848 R11: 0293 R12: 7ffdceb29fa8 [921588.558998] R13: 7ffdceb29fa0 R14: 55ced34b0fa0 R15: 55ced34bcf90 [921588.559014] INFO: task mysqld:1505 blocked for more than 120 seconds. [921588.559018] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.559022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.559025] task:mysqld state:D stack:0 pid: 1505 ppid: 1 flags:0x [921588.559030] Call Trace: [921588.559032] __schedule+0x44c/0x8a0 [921588.559036] schedule+0x4f/0xc0 [921588.559040] md_write_start+0x150/0x240 [921588.559044] ? wait_woken+0x80/0x80 [921588.559047] raid5_make_request+0x88/0x890 [raid456] [921588.559056] ? wait_woken+0x80/0x80 [921588.559059] ? mempool_kmalloc+0x17/0x20 [921588.559062] md_handle_request+0x12d/0x1a0 [921588.559065] ? __part_start_io_acct+0x51/0xf0 [921588.559068] md_submit_bio+0xca/0x100 [921588.559071] submit_bio_noacct+0x112/0x4f0 [921588.559075] submit_bio+0x51/0x1a0 [921588.559077] ? _cond_resched+0x19/0x30 [921588.559081] ? kmem_cache_alloc+0x38e/0x440 [921588.559084] ? ext4_init_io_end+0x1f/0x50 [921588.559088] ext4_io_submit+0x4d/0x60 [921588.559091] ext4_writepages+0x2c6/0xcd0 [921588.559094] ?
[Kernel-packages] [Bug 1942935] Card0.Codecs.codec97.0.ac97.0-0.txt
apport information ** Attachment added: "Card0.Codecs.codec97.0.ac97.0-0.txt" https://bugs.launchpad.net/bugs/1942935/+attachment/5526031/+files/Card0.Codecs.codec97.0.ac97.0-0.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-signed-hwe-5.11 in Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync Status in linux package in Ubuntu: Confirmed Status in linux-signed-hwe-5.11 package in Ubuntu: Confirmed Bug description: It seems to always occur during an mdcheck/resync, if I am logged in via SSH it is still somewhat responsive and basic utilities like dmesg will work. But it apppears any write I/O will hang the terminal and nothing is written to syslog (presumably because it is blocked). Below is output of dmesg and cat /proc/mdstat, it appears the data check was interrupted and /proc/mdstat still shows progress, and a whole slew of hung tasks including md1_resync itself. [756484.534293] md: data-check of RAID array md0 [756484.628039] md: delaying data-check of md1 until md0 has finished (they share one or more physical units) [756493.808773] md: md0: data-check done. [756493.829760] md: data-check of RAID array md1 [778112.446410] md: md1: data-check interrupted. [810654.608102] md: data-check of RAID array md1 [832291.201064] md: md1: data-check interrupted. [899745.389485] md: data-check of RAID array md1 [921395.835305] md: md1: data-check interrupted. [921588.558834] INFO: task systemd-journal:376 blocked for more than 120 seconds. [921588.558846] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.558850] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.558854] task:systemd-journal state:D stack:0 pid: 376 ppid: 1 flags:0x0220 [921588.558859] Call Trace: [921588.558864] __schedule+0x44c/0x8a0 [921588.558872] schedule+0x4f/0xc0 [921588.558876] md_write_start+0x150/0x240 [921588.558880] ? wait_woken+0x80/0x80 [921588.558886] raid5_make_request+0x88/0x890 [raid456] [921588.558898] ? wait_woken+0x80/0x80 [921588.558901] ? mempool_kmalloc+0x17/0x20 [921588.558904] md_handle_request+0x12d/0x1a0 [921588.558907] ? __part_start_io_acct+0x51/0xf0 [921588.558912] md_submit_bio+0xca/0x100 [921588.558915] submit_bio_noacct+0x112/0x4f0 [921588.558918] ? ext4_fc_reserve_space+0x110/0x230 [921588.558922] submit_bio+0x51/0x1a0 [921588.558925] ? _cond_resched+0x19/0x30 [921588.558928] ? kmem_cache_alloc+0x38e/0x440 [921588.558932] ? ext4_init_io_end+0x1f/0x50 [921588.558936] ext4_io_submit+0x4d/0x60 [921588.558940] ext4_writepages+0x2c6/0xcd0 [921588.558944] do_writepages+0x43/0xd0 [921588.558948] ? do_writepages+0x43/0xd0 [921588.558951] ? fault_dirty_shared_page+0xa5/0x110 [921588.558955] __filemap_fdatawrite_range+0xcc/0x110 [921588.558960] file_write_and_wait_range+0x74/0xc0 [921588.558962] ext4_sync_file+0xf5/0x350 [921588.558967] vfs_fsync_range+0x49/0x80 [921588.558970] do_fsync+0x3d/0x70 [921588.558973] __x64_sys_fsync+0x14/0x20 [921588.558976] do_syscall_64+0x38/0x90 [921588.558980] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921588.558984] RIP: 0033:0x7f4c97ee832b [921588.558987] RSP: 002b:7ffdceb29e50 EFLAGS: 0293 ORIG_RAX: 004a [921588.558991] RAX: ffda RBX: 55ced34b0fa0 RCX: 7f4c97ee832b [921588.558993] RDX: 7f4c97fc8000 RSI: 55ced3487b70 RDI: 0021 [921588.558995] RBP: 0001 R08: R09: 7ffdceb29fa8 [921588.558996] R10: 7f4c97d2c848 R11: 0293 R12: 7ffdceb29fa8 [921588.558998] R13: 7ffdceb29fa0 R14: 55ced34b0fa0 R15: 55ced34bcf90 [921588.559014] INFO: task mysqld:1505 blocked for more than 120 seconds. [921588.559018] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.559022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.559025] task:mysqld state:D stack:0 pid: 1505 ppid: 1 flags:0x [921588.559030] Call Trace: [921588.559032] __schedule+0x44c/0x8a0 [921588.559036] schedule+0x4f/0xc0 [921588.559040] md_write_start+0x150/0x240 [921588.559044] ? wait_woken+0x80/0x80 [921588.559047] raid5_make_request+0x88/0x890 [raid456] [921588.559056] ? wait_woken+0x80/0x80 [921588.559059] ? mempool_kmalloc+0x17/0x20 [921588.559062] md_handle_request+0x12d/0x1a0 [921588.559065] ? __part_start_io_acct+0x51/0xf0 [921588.559068] md_submit_bio+0xca/0x100 [921588.559071] submit_bio_noacct+0x112/0x4f0 [921588.559075] submit_bio+0x51/0x1a0 [921588.559077] ? _cond_resched+0x19/0x30 [921588.559081] ? kmem_cache_alloc+0x38e/0x440 [921588.559084] ? ext4_init_io_end+0x1f/0x50 [921588.559088] ext4_io_submit+0x4d/0x60 [921588.559091]
[Kernel-packages] [Bug 1942935] Card0.Codecs.codec97.0.ac97.0-0.regs.txt
apport information ** Attachment added: "Card0.Codecs.codec97.0.ac97.0-0.regs.txt" https://bugs.launchpad.net/bugs/1942935/+attachment/5526032/+files/Card0.Codecs.codec97.0.ac97.0-0.regs.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-signed-hwe-5.11 in Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync Status in linux package in Ubuntu: Confirmed Status in linux-signed-hwe-5.11 package in Ubuntu: Confirmed Bug description: It seems to always occur during an mdcheck/resync, if I am logged in via SSH it is still somewhat responsive and basic utilities like dmesg will work. But it apppears any write I/O will hang the terminal and nothing is written to syslog (presumably because it is blocked). Below is output of dmesg and cat /proc/mdstat, it appears the data check was interrupted and /proc/mdstat still shows progress, and a whole slew of hung tasks including md1_resync itself. [756484.534293] md: data-check of RAID array md0 [756484.628039] md: delaying data-check of md1 until md0 has finished (they share one or more physical units) [756493.808773] md: md0: data-check done. [756493.829760] md: data-check of RAID array md1 [778112.446410] md: md1: data-check interrupted. [810654.608102] md: data-check of RAID array md1 [832291.201064] md: md1: data-check interrupted. [899745.389485] md: data-check of RAID array md1 [921395.835305] md: md1: data-check interrupted. [921588.558834] INFO: task systemd-journal:376 blocked for more than 120 seconds. [921588.558846] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.558850] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.558854] task:systemd-journal state:D stack:0 pid: 376 ppid: 1 flags:0x0220 [921588.558859] Call Trace: [921588.558864] __schedule+0x44c/0x8a0 [921588.558872] schedule+0x4f/0xc0 [921588.558876] md_write_start+0x150/0x240 [921588.558880] ? wait_woken+0x80/0x80 [921588.558886] raid5_make_request+0x88/0x890 [raid456] [921588.558898] ? wait_woken+0x80/0x80 [921588.558901] ? mempool_kmalloc+0x17/0x20 [921588.558904] md_handle_request+0x12d/0x1a0 [921588.558907] ? __part_start_io_acct+0x51/0xf0 [921588.558912] md_submit_bio+0xca/0x100 [921588.558915] submit_bio_noacct+0x112/0x4f0 [921588.558918] ? ext4_fc_reserve_space+0x110/0x230 [921588.558922] submit_bio+0x51/0x1a0 [921588.558925] ? _cond_resched+0x19/0x30 [921588.558928] ? kmem_cache_alloc+0x38e/0x440 [921588.558932] ? ext4_init_io_end+0x1f/0x50 [921588.558936] ext4_io_submit+0x4d/0x60 [921588.558940] ext4_writepages+0x2c6/0xcd0 [921588.558944] do_writepages+0x43/0xd0 [921588.558948] ? do_writepages+0x43/0xd0 [921588.558951] ? fault_dirty_shared_page+0xa5/0x110 [921588.558955] __filemap_fdatawrite_range+0xcc/0x110 [921588.558960] file_write_and_wait_range+0x74/0xc0 [921588.558962] ext4_sync_file+0xf5/0x350 [921588.558967] vfs_fsync_range+0x49/0x80 [921588.558970] do_fsync+0x3d/0x70 [921588.558973] __x64_sys_fsync+0x14/0x20 [921588.558976] do_syscall_64+0x38/0x90 [921588.558980] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921588.558984] RIP: 0033:0x7f4c97ee832b [921588.558987] RSP: 002b:7ffdceb29e50 EFLAGS: 0293 ORIG_RAX: 004a [921588.558991] RAX: ffda RBX: 55ced34b0fa0 RCX: 7f4c97ee832b [921588.558993] RDX: 7f4c97fc8000 RSI: 55ced3487b70 RDI: 0021 [921588.558995] RBP: 0001 R08: R09: 7ffdceb29fa8 [921588.558996] R10: 7f4c97d2c848 R11: 0293 R12: 7ffdceb29fa8 [921588.558998] R13: 7ffdceb29fa0 R14: 55ced34b0fa0 R15: 55ced34bcf90 [921588.559014] INFO: task mysqld:1505 blocked for more than 120 seconds. [921588.559018] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.559022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.559025] task:mysqld state:D stack:0 pid: 1505 ppid: 1 flags:0x [921588.559030] Call Trace: [921588.559032] __schedule+0x44c/0x8a0 [921588.559036] schedule+0x4f/0xc0 [921588.559040] md_write_start+0x150/0x240 [921588.559044] ? wait_woken+0x80/0x80 [921588.559047] raid5_make_request+0x88/0x890 [raid456] [921588.559056] ? wait_woken+0x80/0x80 [921588.559059] ? mempool_kmalloc+0x17/0x20 [921588.559062] md_handle_request+0x12d/0x1a0 [921588.559065] ? __part_start_io_acct+0x51/0xf0 [921588.559068] md_submit_bio+0xca/0x100 [921588.559071] submit_bio_noacct+0x112/0x4f0 [921588.559075] submit_bio+0x51/0x1a0 [921588.559077] ? _cond_resched+0x19/0x30 [921588.559081] ? kmem_cache_alloc+0x38e/0x440 [921588.559084] ? ext4_init_io_end+0x1f/0x50 [921588.559088] ext4_io_submit+0x4d/0x60 [921588.559091]
[Kernel-packages] [Bug 1942935] CurrentDmesg.txt
apport information ** Attachment added: "CurrentDmesg.txt" https://bugs.launchpad.net/bugs/1942935/+attachment/5526033/+files/CurrentDmesg.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-signed-hwe-5.11 in Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync Status in linux package in Ubuntu: Confirmed Status in linux-signed-hwe-5.11 package in Ubuntu: Confirmed Bug description: It seems to always occur during an mdcheck/resync, if I am logged in via SSH it is still somewhat responsive and basic utilities like dmesg will work. But it apppears any write I/O will hang the terminal and nothing is written to syslog (presumably because it is blocked). Below is output of dmesg and cat /proc/mdstat, it appears the data check was interrupted and /proc/mdstat still shows progress, and a whole slew of hung tasks including md1_resync itself. [756484.534293] md: data-check of RAID array md0 [756484.628039] md: delaying data-check of md1 until md0 has finished (they share one or more physical units) [756493.808773] md: md0: data-check done. [756493.829760] md: data-check of RAID array md1 [778112.446410] md: md1: data-check interrupted. [810654.608102] md: data-check of RAID array md1 [832291.201064] md: md1: data-check interrupted. [899745.389485] md: data-check of RAID array md1 [921395.835305] md: md1: data-check interrupted. [921588.558834] INFO: task systemd-journal:376 blocked for more than 120 seconds. [921588.558846] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.558850] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.558854] task:systemd-journal state:D stack:0 pid: 376 ppid: 1 flags:0x0220 [921588.558859] Call Trace: [921588.558864] __schedule+0x44c/0x8a0 [921588.558872] schedule+0x4f/0xc0 [921588.558876] md_write_start+0x150/0x240 [921588.558880] ? wait_woken+0x80/0x80 [921588.558886] raid5_make_request+0x88/0x890 [raid456] [921588.558898] ? wait_woken+0x80/0x80 [921588.558901] ? mempool_kmalloc+0x17/0x20 [921588.558904] md_handle_request+0x12d/0x1a0 [921588.558907] ? __part_start_io_acct+0x51/0xf0 [921588.558912] md_submit_bio+0xca/0x100 [921588.558915] submit_bio_noacct+0x112/0x4f0 [921588.558918] ? ext4_fc_reserve_space+0x110/0x230 [921588.558922] submit_bio+0x51/0x1a0 [921588.558925] ? _cond_resched+0x19/0x30 [921588.558928] ? kmem_cache_alloc+0x38e/0x440 [921588.558932] ? ext4_init_io_end+0x1f/0x50 [921588.558936] ext4_io_submit+0x4d/0x60 [921588.558940] ext4_writepages+0x2c6/0xcd0 [921588.558944] do_writepages+0x43/0xd0 [921588.558948] ? do_writepages+0x43/0xd0 [921588.558951] ? fault_dirty_shared_page+0xa5/0x110 [921588.558955] __filemap_fdatawrite_range+0xcc/0x110 [921588.558960] file_write_and_wait_range+0x74/0xc0 [921588.558962] ext4_sync_file+0xf5/0x350 [921588.558967] vfs_fsync_range+0x49/0x80 [921588.558970] do_fsync+0x3d/0x70 [921588.558973] __x64_sys_fsync+0x14/0x20 [921588.558976] do_syscall_64+0x38/0x90 [921588.558980] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921588.558984] RIP: 0033:0x7f4c97ee832b [921588.558987] RSP: 002b:7ffdceb29e50 EFLAGS: 0293 ORIG_RAX: 004a [921588.558991] RAX: ffda RBX: 55ced34b0fa0 RCX: 7f4c97ee832b [921588.558993] RDX: 7f4c97fc8000 RSI: 55ced3487b70 RDI: 0021 [921588.558995] RBP: 0001 R08: R09: 7ffdceb29fa8 [921588.558996] R10: 7f4c97d2c848 R11: 0293 R12: 7ffdceb29fa8 [921588.558998] R13: 7ffdceb29fa0 R14: 55ced34b0fa0 R15: 55ced34bcf90 [921588.559014] INFO: task mysqld:1505 blocked for more than 120 seconds. [921588.559018] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.559022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.559025] task:mysqld state:D stack:0 pid: 1505 ppid: 1 flags:0x [921588.559030] Call Trace: [921588.559032] __schedule+0x44c/0x8a0 [921588.559036] schedule+0x4f/0xc0 [921588.559040] md_write_start+0x150/0x240 [921588.559044] ? wait_woken+0x80/0x80 [921588.559047] raid5_make_request+0x88/0x890 [raid456] [921588.559056] ? wait_woken+0x80/0x80 [921588.559059] ? mempool_kmalloc+0x17/0x20 [921588.559062] md_handle_request+0x12d/0x1a0 [921588.559065] ? __part_start_io_acct+0x51/0xf0 [921588.559068] md_submit_bio+0xca/0x100 [921588.559071] submit_bio_noacct+0x112/0x4f0 [921588.559075] submit_bio+0x51/0x1a0 [921588.559077] ? _cond_resched+0x19/0x30 [921588.559081] ? kmem_cache_alloc+0x38e/0x440 [921588.559084] ? ext4_init_io_end+0x1f/0x50 [921588.559088] ext4_io_submit+0x4d/0x60 [921588.559091] ext4_writepages+0x2c6/0xcd0 [921588.559094] ?
[Kernel-packages] [Bug 1942935] CRDA.txt
apport information ** Attachment added: "CRDA.txt" https://bugs.launchpad.net/bugs/1942935/+attachment/5526030/+files/CRDA.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-signed-hwe-5.11 in Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync Status in linux package in Ubuntu: Confirmed Status in linux-signed-hwe-5.11 package in Ubuntu: Confirmed Bug description: It seems to always occur during an mdcheck/resync, if I am logged in via SSH it is still somewhat responsive and basic utilities like dmesg will work. But it apppears any write I/O will hang the terminal and nothing is written to syslog (presumably because it is blocked). Below is output of dmesg and cat /proc/mdstat, it appears the data check was interrupted and /proc/mdstat still shows progress, and a whole slew of hung tasks including md1_resync itself. [756484.534293] md: data-check of RAID array md0 [756484.628039] md: delaying data-check of md1 until md0 has finished (they share one or more physical units) [756493.808773] md: md0: data-check done. [756493.829760] md: data-check of RAID array md1 [778112.446410] md: md1: data-check interrupted. [810654.608102] md: data-check of RAID array md1 [832291.201064] md: md1: data-check interrupted. [899745.389485] md: data-check of RAID array md1 [921395.835305] md: md1: data-check interrupted. [921588.558834] INFO: task systemd-journal:376 blocked for more than 120 seconds. [921588.558846] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.558850] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.558854] task:systemd-journal state:D stack:0 pid: 376 ppid: 1 flags:0x0220 [921588.558859] Call Trace: [921588.558864] __schedule+0x44c/0x8a0 [921588.558872] schedule+0x4f/0xc0 [921588.558876] md_write_start+0x150/0x240 [921588.558880] ? wait_woken+0x80/0x80 [921588.558886] raid5_make_request+0x88/0x890 [raid456] [921588.558898] ? wait_woken+0x80/0x80 [921588.558901] ? mempool_kmalloc+0x17/0x20 [921588.558904] md_handle_request+0x12d/0x1a0 [921588.558907] ? __part_start_io_acct+0x51/0xf0 [921588.558912] md_submit_bio+0xca/0x100 [921588.558915] submit_bio_noacct+0x112/0x4f0 [921588.558918] ? ext4_fc_reserve_space+0x110/0x230 [921588.558922] submit_bio+0x51/0x1a0 [921588.558925] ? _cond_resched+0x19/0x30 [921588.558928] ? kmem_cache_alloc+0x38e/0x440 [921588.558932] ? ext4_init_io_end+0x1f/0x50 [921588.558936] ext4_io_submit+0x4d/0x60 [921588.558940] ext4_writepages+0x2c6/0xcd0 [921588.558944] do_writepages+0x43/0xd0 [921588.558948] ? do_writepages+0x43/0xd0 [921588.558951] ? fault_dirty_shared_page+0xa5/0x110 [921588.558955] __filemap_fdatawrite_range+0xcc/0x110 [921588.558960] file_write_and_wait_range+0x74/0xc0 [921588.558962] ext4_sync_file+0xf5/0x350 [921588.558967] vfs_fsync_range+0x49/0x80 [921588.558970] do_fsync+0x3d/0x70 [921588.558973] __x64_sys_fsync+0x14/0x20 [921588.558976] do_syscall_64+0x38/0x90 [921588.558980] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921588.558984] RIP: 0033:0x7f4c97ee832b [921588.558987] RSP: 002b:7ffdceb29e50 EFLAGS: 0293 ORIG_RAX: 004a [921588.558991] RAX: ffda RBX: 55ced34b0fa0 RCX: 7f4c97ee832b [921588.558993] RDX: 7f4c97fc8000 RSI: 55ced3487b70 RDI: 0021 [921588.558995] RBP: 0001 R08: R09: 7ffdceb29fa8 [921588.558996] R10: 7f4c97d2c848 R11: 0293 R12: 7ffdceb29fa8 [921588.558998] R13: 7ffdceb29fa0 R14: 55ced34b0fa0 R15: 55ced34bcf90 [921588.559014] INFO: task mysqld:1505 blocked for more than 120 seconds. [921588.559018] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.559022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.559025] task:mysqld state:D stack:0 pid: 1505 ppid: 1 flags:0x [921588.559030] Call Trace: [921588.559032] __schedule+0x44c/0x8a0 [921588.559036] schedule+0x4f/0xc0 [921588.559040] md_write_start+0x150/0x240 [921588.559044] ? wait_woken+0x80/0x80 [921588.559047] raid5_make_request+0x88/0x890 [raid456] [921588.559056] ? wait_woken+0x80/0x80 [921588.559059] ? mempool_kmalloc+0x17/0x20 [921588.559062] md_handle_request+0x12d/0x1a0 [921588.559065] ? __part_start_io_acct+0x51/0xf0 [921588.559068] md_submit_bio+0xca/0x100 [921588.559071] submit_bio_noacct+0x112/0x4f0 [921588.559075] submit_bio+0x51/0x1a0 [921588.559077] ? _cond_resched+0x19/0x30 [921588.559081] ? kmem_cache_alloc+0x38e/0x440 [921588.559084] ? ext4_init_io_end+0x1f/0x50 [921588.559088] ext4_io_submit+0x4d/0x60 [921588.559091] ext4_writepages+0x2c6/0xcd0 [921588.559094] ? __schedule+0x454/0x8a0
[Kernel-packages] [Bug 1942935] AlsaDevices.txt
apport information ** Attachment added: "AlsaDevices.txt" https://bugs.launchpad.net/bugs/1942935/+attachment/5526029/+files/AlsaDevices.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-signed-hwe-5.11 in Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync Status in linux package in Ubuntu: Confirmed Status in linux-signed-hwe-5.11 package in Ubuntu: Confirmed Bug description: It seems to always occur during an mdcheck/resync, if I am logged in via SSH it is still somewhat responsive and basic utilities like dmesg will work. But it apppears any write I/O will hang the terminal and nothing is written to syslog (presumably because it is blocked). Below is output of dmesg and cat /proc/mdstat, it appears the data check was interrupted and /proc/mdstat still shows progress, and a whole slew of hung tasks including md1_resync itself. [756484.534293] md: data-check of RAID array md0 [756484.628039] md: delaying data-check of md1 until md0 has finished (they share one or more physical units) [756493.808773] md: md0: data-check done. [756493.829760] md: data-check of RAID array md1 [778112.446410] md: md1: data-check interrupted. [810654.608102] md: data-check of RAID array md1 [832291.201064] md: md1: data-check interrupted. [899745.389485] md: data-check of RAID array md1 [921395.835305] md: md1: data-check interrupted. [921588.558834] INFO: task systemd-journal:376 blocked for more than 120 seconds. [921588.558846] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.558850] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.558854] task:systemd-journal state:D stack:0 pid: 376 ppid: 1 flags:0x0220 [921588.558859] Call Trace: [921588.558864] __schedule+0x44c/0x8a0 [921588.558872] schedule+0x4f/0xc0 [921588.558876] md_write_start+0x150/0x240 [921588.558880] ? wait_woken+0x80/0x80 [921588.558886] raid5_make_request+0x88/0x890 [raid456] [921588.558898] ? wait_woken+0x80/0x80 [921588.558901] ? mempool_kmalloc+0x17/0x20 [921588.558904] md_handle_request+0x12d/0x1a0 [921588.558907] ? __part_start_io_acct+0x51/0xf0 [921588.558912] md_submit_bio+0xca/0x100 [921588.558915] submit_bio_noacct+0x112/0x4f0 [921588.558918] ? ext4_fc_reserve_space+0x110/0x230 [921588.558922] submit_bio+0x51/0x1a0 [921588.558925] ? _cond_resched+0x19/0x30 [921588.558928] ? kmem_cache_alloc+0x38e/0x440 [921588.558932] ? ext4_init_io_end+0x1f/0x50 [921588.558936] ext4_io_submit+0x4d/0x60 [921588.558940] ext4_writepages+0x2c6/0xcd0 [921588.558944] do_writepages+0x43/0xd0 [921588.558948] ? do_writepages+0x43/0xd0 [921588.558951] ? fault_dirty_shared_page+0xa5/0x110 [921588.558955] __filemap_fdatawrite_range+0xcc/0x110 [921588.558960] file_write_and_wait_range+0x74/0xc0 [921588.558962] ext4_sync_file+0xf5/0x350 [921588.558967] vfs_fsync_range+0x49/0x80 [921588.558970] do_fsync+0x3d/0x70 [921588.558973] __x64_sys_fsync+0x14/0x20 [921588.558976] do_syscall_64+0x38/0x90 [921588.558980] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921588.558984] RIP: 0033:0x7f4c97ee832b [921588.558987] RSP: 002b:7ffdceb29e50 EFLAGS: 0293 ORIG_RAX: 004a [921588.558991] RAX: ffda RBX: 55ced34b0fa0 RCX: 7f4c97ee832b [921588.558993] RDX: 7f4c97fc8000 RSI: 55ced3487b70 RDI: 0021 [921588.558995] RBP: 0001 R08: R09: 7ffdceb29fa8 [921588.558996] R10: 7f4c97d2c848 R11: 0293 R12: 7ffdceb29fa8 [921588.558998] R13: 7ffdceb29fa0 R14: 55ced34b0fa0 R15: 55ced34bcf90 [921588.559014] INFO: task mysqld:1505 blocked for more than 120 seconds. [921588.559018] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.559022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.559025] task:mysqld state:D stack:0 pid: 1505 ppid: 1 flags:0x [921588.559030] Call Trace: [921588.559032] __schedule+0x44c/0x8a0 [921588.559036] schedule+0x4f/0xc0 [921588.559040] md_write_start+0x150/0x240 [921588.559044] ? wait_woken+0x80/0x80 [921588.559047] raid5_make_request+0x88/0x890 [raid456] [921588.559056] ? wait_woken+0x80/0x80 [921588.559059] ? mempool_kmalloc+0x17/0x20 [921588.559062] md_handle_request+0x12d/0x1a0 [921588.559065] ? __part_start_io_acct+0x51/0xf0 [921588.559068] md_submit_bio+0xca/0x100 [921588.559071] submit_bio_noacct+0x112/0x4f0 [921588.559075] submit_bio+0x51/0x1a0 [921588.559077] ? _cond_resched+0x19/0x30 [921588.559081] ? kmem_cache_alloc+0x38e/0x440 [921588.559084] ? ext4_init_io_end+0x1f/0x50 [921588.559088] ext4_io_submit+0x4d/0x60 [921588.559091] ext4_writepages+0x2c6/0xcd0 [921588.559094] ?
[Kernel-packages] [Bug 1942935] Re: kernel io hangs during mdcheck/resync
** Patch added: "md-reap-sync-thread.patch" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942935/+attachment/5526028/+files/md-reap-sync-thread.patch ** Tags added: apport-collected impish ** Description changed: It seems to always occur during an mdcheck/resync, if I am logged in via SSH it is still somewhat responsive and basic utilities like dmesg will work. But it apppears any write I/O will hang the terminal and nothing is written to syslog (presumably because it is blocked). Below is output of dmesg and cat /proc/mdstat, it appears the data check was interrupted and /proc/mdstat still shows progress, and a whole slew of hung tasks including md1_resync itself. [756484.534293] md: data-check of RAID array md0 [756484.628039] md: delaying data-check of md1 until md0 has finished (they share one or more physical units) [756493.808773] md: md0: data-check done. [756493.829760] md: data-check of RAID array md1 [778112.446410] md: md1: data-check interrupted. [810654.608102] md: data-check of RAID array md1 [832291.201064] md: md1: data-check interrupted. [899745.389485] md: data-check of RAID array md1 [921395.835305] md: md1: data-check interrupted. [921588.558834] INFO: task systemd-journal:376 blocked for more than 120 seconds. [921588.558846] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.558850] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.558854] task:systemd-journal state:D stack:0 pid: 376 ppid: 1 flags:0x0220 [921588.558859] Call Trace: [921588.558864] __schedule+0x44c/0x8a0 [921588.558872] schedule+0x4f/0xc0 [921588.558876] md_write_start+0x150/0x240 [921588.558880] ? wait_woken+0x80/0x80 [921588.558886] raid5_make_request+0x88/0x890 [raid456] [921588.558898] ? wait_woken+0x80/0x80 [921588.558901] ? mempool_kmalloc+0x17/0x20 [921588.558904] md_handle_request+0x12d/0x1a0 [921588.558907] ? __part_start_io_acct+0x51/0xf0 [921588.558912] md_submit_bio+0xca/0x100 [921588.558915] submit_bio_noacct+0x112/0x4f0 [921588.558918] ? ext4_fc_reserve_space+0x110/0x230 [921588.558922] submit_bio+0x51/0x1a0 [921588.558925] ? _cond_resched+0x19/0x30 [921588.558928] ? kmem_cache_alloc+0x38e/0x440 [921588.558932] ? ext4_init_io_end+0x1f/0x50 [921588.558936] ext4_io_submit+0x4d/0x60 [921588.558940] ext4_writepages+0x2c6/0xcd0 [921588.558944] do_writepages+0x43/0xd0 [921588.558948] ? do_writepages+0x43/0xd0 [921588.558951] ? fault_dirty_shared_page+0xa5/0x110 [921588.558955] __filemap_fdatawrite_range+0xcc/0x110 [921588.558960] file_write_and_wait_range+0x74/0xc0 [921588.558962] ext4_sync_file+0xf5/0x350 [921588.558967] vfs_fsync_range+0x49/0x80 [921588.558970] do_fsync+0x3d/0x70 [921588.558973] __x64_sys_fsync+0x14/0x20 [921588.558976] do_syscall_64+0x38/0x90 [921588.558980] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921588.558984] RIP: 0033:0x7f4c97ee832b [921588.558987] RSP: 002b:7ffdceb29e50 EFLAGS: 0293 ORIG_RAX: 004a [921588.558991] RAX: ffda RBX: 55ced34b0fa0 RCX: 7f4c97ee832b [921588.558993] RDX: 7f4c97fc8000 RSI: 55ced3487b70 RDI: 0021 [921588.558995] RBP: 0001 R08: R09: 7ffdceb29fa8 [921588.558996] R10: 7f4c97d2c848 R11: 0293 R12: 7ffdceb29fa8 [921588.558998] R13: 7ffdceb29fa0 R14: 55ced34b0fa0 R15: 55ced34bcf90 [921588.559014] INFO: task mysqld:1505 blocked for more than 120 seconds. [921588.559018] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.559022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.559025] task:mysqld state:D stack:0 pid: 1505 ppid: 1 flags:0x [921588.559030] Call Trace: [921588.559032] __schedule+0x44c/0x8a0 [921588.559036] schedule+0x4f/0xc0 [921588.559040] md_write_start+0x150/0x240 [921588.559044] ? wait_woken+0x80/0x80 [921588.559047] raid5_make_request+0x88/0x890 [raid456] [921588.559056] ? wait_woken+0x80/0x80 [921588.559059] ? mempool_kmalloc+0x17/0x20 [921588.559062] md_handle_request+0x12d/0x1a0 [921588.559065] ? __part_start_io_acct+0x51/0xf0 [921588.559068] md_submit_bio+0xca/0x100 [921588.559071] submit_bio_noacct+0x112/0x4f0 [921588.559075] submit_bio+0x51/0x1a0 [921588.559077] ? _cond_resched+0x19/0x30 [921588.559081] ? kmem_cache_alloc+0x38e/0x440 [921588.559084] ? ext4_init_io_end+0x1f/0x50 [921588.559088] ext4_io_submit+0x4d/0x60 [921588.559091] ext4_writepages+0x2c6/0xcd0 [921588.559094] ? __schedule+0x454/0x8a0 [921588.559097] ? hrtimer_start_range_ns+0x1aa/0x2f0 [921588.559100] ? timerqueue_del+0x24/0x50 [921588.559105] ? futex_wait+0x1ed/0x270 [921588.559109] do_writepages+0x43/0xd0 [921588.559112] ? do_writepages+0x43/0xd0 [921588.559115] ?
[Kernel-packages] [Bug 1942935] Re: kernel io hangs during mdcheck/resync
** Changed in: linux (Ubuntu) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-signed-hwe-5.11 in Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync Status in linux package in Ubuntu: Confirmed Status in linux-signed-hwe-5.11 package in Ubuntu: Confirmed Bug description: It seems to always occur during an mdcheck/resync, if I am logged in via SSH it is still somewhat responsive and basic utilities like dmesg will work. But it apppears any write I/O will hang the terminal and nothing is written to syslog (presumably because it is blocked). Below is output of dmesg and cat /proc/mdstat, it appears the data check was interrupted and /proc/mdstat still shows progress, and a whole slew of hung tasks including md1_resync itself. [756484.534293] md: data-check of RAID array md0 [756484.628039] md: delaying data-check of md1 until md0 has finished (they share one or more physical units) [756493.808773] md: md0: data-check done. [756493.829760] md: data-check of RAID array md1 [778112.446410] md: md1: data-check interrupted. [810654.608102] md: data-check of RAID array md1 [832291.201064] md: md1: data-check interrupted. [899745.389485] md: data-check of RAID array md1 [921395.835305] md: md1: data-check interrupted. [921588.558834] INFO: task systemd-journal:376 blocked for more than 120 seconds. [921588.558846] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.558850] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.558854] task:systemd-journal state:D stack:0 pid: 376 ppid: 1 flags:0x0220 [921588.558859] Call Trace: [921588.558864] __schedule+0x44c/0x8a0 [921588.558872] schedule+0x4f/0xc0 [921588.558876] md_write_start+0x150/0x240 [921588.558880] ? wait_woken+0x80/0x80 [921588.558886] raid5_make_request+0x88/0x890 [raid456] [921588.558898] ? wait_woken+0x80/0x80 [921588.558901] ? mempool_kmalloc+0x17/0x20 [921588.558904] md_handle_request+0x12d/0x1a0 [921588.558907] ? __part_start_io_acct+0x51/0xf0 [921588.558912] md_submit_bio+0xca/0x100 [921588.558915] submit_bio_noacct+0x112/0x4f0 [921588.558918] ? ext4_fc_reserve_space+0x110/0x230 [921588.558922] submit_bio+0x51/0x1a0 [921588.558925] ? _cond_resched+0x19/0x30 [921588.558928] ? kmem_cache_alloc+0x38e/0x440 [921588.558932] ? ext4_init_io_end+0x1f/0x50 [921588.558936] ext4_io_submit+0x4d/0x60 [921588.558940] ext4_writepages+0x2c6/0xcd0 [921588.558944] do_writepages+0x43/0xd0 [921588.558948] ? do_writepages+0x43/0xd0 [921588.558951] ? fault_dirty_shared_page+0xa5/0x110 [921588.558955] __filemap_fdatawrite_range+0xcc/0x110 [921588.558960] file_write_and_wait_range+0x74/0xc0 [921588.558962] ext4_sync_file+0xf5/0x350 [921588.558967] vfs_fsync_range+0x49/0x80 [921588.558970] do_fsync+0x3d/0x70 [921588.558973] __x64_sys_fsync+0x14/0x20 [921588.558976] do_syscall_64+0x38/0x90 [921588.558980] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921588.558984] RIP: 0033:0x7f4c97ee832b [921588.558987] RSP: 002b:7ffdceb29e50 EFLAGS: 0293 ORIG_RAX: 004a [921588.558991] RAX: ffda RBX: 55ced34b0fa0 RCX: 7f4c97ee832b [921588.558993] RDX: 7f4c97fc8000 RSI: 55ced3487b70 RDI: 0021 [921588.558995] RBP: 0001 R08: R09: 7ffdceb29fa8 [921588.558996] R10: 7f4c97d2c848 R11: 0293 R12: 7ffdceb29fa8 [921588.558998] R13: 7ffdceb29fa0 R14: 55ced34b0fa0 R15: 55ced34bcf90 [921588.559014] INFO: task mysqld:1505 blocked for more than 120 seconds. [921588.559018] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.559022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.559025] task:mysqld state:D stack:0 pid: 1505 ppid: 1 flags:0x [921588.559030] Call Trace: [921588.559032] __schedule+0x44c/0x8a0 [921588.559036] schedule+0x4f/0xc0 [921588.559040] md_write_start+0x150/0x240 [921588.559044] ? wait_woken+0x80/0x80 [921588.559047] raid5_make_request+0x88/0x890 [raid456] [921588.559056] ? wait_woken+0x80/0x80 [921588.559059] ? mempool_kmalloc+0x17/0x20 [921588.559062] md_handle_request+0x12d/0x1a0 [921588.559065] ? __part_start_io_acct+0x51/0xf0 [921588.559068] md_submit_bio+0xca/0x100 [921588.559071] submit_bio_noacct+0x112/0x4f0 [921588.559075] submit_bio+0x51/0x1a0 [921588.559077] ? _cond_resched+0x19/0x30 [921588.559081] ? kmem_cache_alloc+0x38e/0x440 [921588.559084] ? ext4_init_io_end+0x1f/0x50 [921588.559088] ext4_io_submit+0x4d/0x60 [921588.559091] ext4_writepages+0x2c6/0xcd0 [921588.559094] ? __schedule+0x454/0x8a0 [921588.559097] ? hrtimer_start_range_ns+0x1aa/0x2f0
[Kernel-packages] [Bug 1942935] Re: kernel io hangs during mdcheck/resync
** Tags removed: hirsute -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-signed-hwe-5.11 in Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync Status in linux package in Ubuntu: Incomplete Status in linux-signed-hwe-5.11 package in Ubuntu: Confirmed Bug description: It seems to always occur during an mdcheck/resync, if I am logged in via SSH it is still somewhat responsive and basic utilities like dmesg will work. But it apppears any write I/O will hang the terminal and nothing is written to syslog (presumably because it is blocked). Below is output of dmesg and cat /proc/mdstat, it appears the data check was interrupted and /proc/mdstat still shows progress, and a whole slew of hung tasks including md1_resync itself. [756484.534293] md: data-check of RAID array md0 [756484.628039] md: delaying data-check of md1 until md0 has finished (they share one or more physical units) [756493.808773] md: md0: data-check done. [756493.829760] md: data-check of RAID array md1 [778112.446410] md: md1: data-check interrupted. [810654.608102] md: data-check of RAID array md1 [832291.201064] md: md1: data-check interrupted. [899745.389485] md: data-check of RAID array md1 [921395.835305] md: md1: data-check interrupted. [921588.558834] INFO: task systemd-journal:376 blocked for more than 120 seconds. [921588.558846] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.558850] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.558854] task:systemd-journal state:D stack:0 pid: 376 ppid: 1 flags:0x0220 [921588.558859] Call Trace: [921588.558864] __schedule+0x44c/0x8a0 [921588.558872] schedule+0x4f/0xc0 [921588.558876] md_write_start+0x150/0x240 [921588.558880] ? wait_woken+0x80/0x80 [921588.558886] raid5_make_request+0x88/0x890 [raid456] [921588.558898] ? wait_woken+0x80/0x80 [921588.558901] ? mempool_kmalloc+0x17/0x20 [921588.558904] md_handle_request+0x12d/0x1a0 [921588.558907] ? __part_start_io_acct+0x51/0xf0 [921588.558912] md_submit_bio+0xca/0x100 [921588.558915] submit_bio_noacct+0x112/0x4f0 [921588.558918] ? ext4_fc_reserve_space+0x110/0x230 [921588.558922] submit_bio+0x51/0x1a0 [921588.558925] ? _cond_resched+0x19/0x30 [921588.558928] ? kmem_cache_alloc+0x38e/0x440 [921588.558932] ? ext4_init_io_end+0x1f/0x50 [921588.558936] ext4_io_submit+0x4d/0x60 [921588.558940] ext4_writepages+0x2c6/0xcd0 [921588.558944] do_writepages+0x43/0xd0 [921588.558948] ? do_writepages+0x43/0xd0 [921588.558951] ? fault_dirty_shared_page+0xa5/0x110 [921588.558955] __filemap_fdatawrite_range+0xcc/0x110 [921588.558960] file_write_and_wait_range+0x74/0xc0 [921588.558962] ext4_sync_file+0xf5/0x350 [921588.558967] vfs_fsync_range+0x49/0x80 [921588.558970] do_fsync+0x3d/0x70 [921588.558973] __x64_sys_fsync+0x14/0x20 [921588.558976] do_syscall_64+0x38/0x90 [921588.558980] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921588.558984] RIP: 0033:0x7f4c97ee832b [921588.558987] RSP: 002b:7ffdceb29e50 EFLAGS: 0293 ORIG_RAX: 004a [921588.558991] RAX: ffda RBX: 55ced34b0fa0 RCX: 7f4c97ee832b [921588.558993] RDX: 7f4c97fc8000 RSI: 55ced3487b70 RDI: 0021 [921588.558995] RBP: 0001 R08: R09: 7ffdceb29fa8 [921588.558996] R10: 7f4c97d2c848 R11: 0293 R12: 7ffdceb29fa8 [921588.558998] R13: 7ffdceb29fa0 R14: 55ced34b0fa0 R15: 55ced34bcf90 [921588.559014] INFO: task mysqld:1505 blocked for more than 120 seconds. [921588.559018] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.559022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.559025] task:mysqld state:D stack:0 pid: 1505 ppid: 1 flags:0x [921588.559030] Call Trace: [921588.559032] __schedule+0x44c/0x8a0 [921588.559036] schedule+0x4f/0xc0 [921588.559040] md_write_start+0x150/0x240 [921588.559044] ? wait_woken+0x80/0x80 [921588.559047] raid5_make_request+0x88/0x890 [raid456] [921588.559056] ? wait_woken+0x80/0x80 [921588.559059] ? mempool_kmalloc+0x17/0x20 [921588.559062] md_handle_request+0x12d/0x1a0 [921588.559065] ? __part_start_io_acct+0x51/0xf0 [921588.559068] md_submit_bio+0xca/0x100 [921588.559071] submit_bio_noacct+0x112/0x4f0 [921588.559075] submit_bio+0x51/0x1a0 [921588.559077] ? _cond_resched+0x19/0x30 [921588.559081] ? kmem_cache_alloc+0x38e/0x440 [921588.559084] ? ext4_init_io_end+0x1f/0x50 [921588.559088] ext4_io_submit+0x4d/0x60 [921588.559091] ext4_writepages+0x2c6/0xcd0 [921588.559094] ? __schedule+0x454/0x8a0 [921588.559097] ? hrtimer_start_range_ns+0x1aa/0x2f0 [921588.559100] ? timerqueue_del+0x24/0x50
[Kernel-packages] [Bug 1942935] Re: kernel io hangs during mdcheck/resync
** Also affects: linux (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-signed-hwe-5.11 in Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync Status in linux package in Ubuntu: Incomplete Status in linux-signed-hwe-5.11 package in Ubuntu: Confirmed Bug description: It seems to always occur during an mdcheck/resync, if I am logged in via SSH it is still somewhat responsive and basic utilities like dmesg will work. But it apppears any write I/O will hang the terminal and nothing is written to syslog (presumably because it is blocked). Below is output of dmesg and cat /proc/mdstat, it appears the data check was interrupted and /proc/mdstat still shows progress, and a whole slew of hung tasks including md1_resync itself. [756484.534293] md: data-check of RAID array md0 [756484.628039] md: delaying data-check of md1 until md0 has finished (they share one or more physical units) [756493.808773] md: md0: data-check done. [756493.829760] md: data-check of RAID array md1 [778112.446410] md: md1: data-check interrupted. [810654.608102] md: data-check of RAID array md1 [832291.201064] md: md1: data-check interrupted. [899745.389485] md: data-check of RAID array md1 [921395.835305] md: md1: data-check interrupted. [921588.558834] INFO: task systemd-journal:376 blocked for more than 120 seconds. [921588.558846] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.558850] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.558854] task:systemd-journal state:D stack:0 pid: 376 ppid: 1 flags:0x0220 [921588.558859] Call Trace: [921588.558864] __schedule+0x44c/0x8a0 [921588.558872] schedule+0x4f/0xc0 [921588.558876] md_write_start+0x150/0x240 [921588.558880] ? wait_woken+0x80/0x80 [921588.558886] raid5_make_request+0x88/0x890 [raid456] [921588.558898] ? wait_woken+0x80/0x80 [921588.558901] ? mempool_kmalloc+0x17/0x20 [921588.558904] md_handle_request+0x12d/0x1a0 [921588.558907] ? __part_start_io_acct+0x51/0xf0 [921588.558912] md_submit_bio+0xca/0x100 [921588.558915] submit_bio_noacct+0x112/0x4f0 [921588.558918] ? ext4_fc_reserve_space+0x110/0x230 [921588.558922] submit_bio+0x51/0x1a0 [921588.558925] ? _cond_resched+0x19/0x30 [921588.558928] ? kmem_cache_alloc+0x38e/0x440 [921588.558932] ? ext4_init_io_end+0x1f/0x50 [921588.558936] ext4_io_submit+0x4d/0x60 [921588.558940] ext4_writepages+0x2c6/0xcd0 [921588.558944] do_writepages+0x43/0xd0 [921588.558948] ? do_writepages+0x43/0xd0 [921588.558951] ? fault_dirty_shared_page+0xa5/0x110 [921588.558955] __filemap_fdatawrite_range+0xcc/0x110 [921588.558960] file_write_and_wait_range+0x74/0xc0 [921588.558962] ext4_sync_file+0xf5/0x350 [921588.558967] vfs_fsync_range+0x49/0x80 [921588.558970] do_fsync+0x3d/0x70 [921588.558973] __x64_sys_fsync+0x14/0x20 [921588.558976] do_syscall_64+0x38/0x90 [921588.558980] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921588.558984] RIP: 0033:0x7f4c97ee832b [921588.558987] RSP: 002b:7ffdceb29e50 EFLAGS: 0293 ORIG_RAX: 004a [921588.558991] RAX: ffda RBX: 55ced34b0fa0 RCX: 7f4c97ee832b [921588.558993] RDX: 7f4c97fc8000 RSI: 55ced3487b70 RDI: 0021 [921588.558995] RBP: 0001 R08: R09: 7ffdceb29fa8 [921588.558996] R10: 7f4c97d2c848 R11: 0293 R12: 7ffdceb29fa8 [921588.558998] R13: 7ffdceb29fa0 R14: 55ced34b0fa0 R15: 55ced34bcf90 [921588.559014] INFO: task mysqld:1505 blocked for more than 120 seconds. [921588.559018] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.559022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.559025] task:mysqld state:D stack:0 pid: 1505 ppid: 1 flags:0x [921588.559030] Call Trace: [921588.559032] __schedule+0x44c/0x8a0 [921588.559036] schedule+0x4f/0xc0 [921588.559040] md_write_start+0x150/0x240 [921588.559044] ? wait_woken+0x80/0x80 [921588.559047] raid5_make_request+0x88/0x890 [raid456] [921588.559056] ? wait_woken+0x80/0x80 [921588.559059] ? mempool_kmalloc+0x17/0x20 [921588.559062] md_handle_request+0x12d/0x1a0 [921588.559065] ? __part_start_io_acct+0x51/0xf0 [921588.559068] md_submit_bio+0xca/0x100 [921588.559071] submit_bio_noacct+0x112/0x4f0 [921588.559075] submit_bio+0x51/0x1a0 [921588.559077] ? _cond_resched+0x19/0x30 [921588.559081] ? kmem_cache_alloc+0x38e/0x440 [921588.559084] ? ext4_init_io_end+0x1f/0x50 [921588.559088] ext4_io_submit+0x4d/0x60 [921588.559091] ext4_writepages+0x2c6/0xcd0 [921588.559094] ? __schedule+0x454/0x8a0 [921588.559097] ? hrtimer_start_range_ns+0x1aa/0x2f0
[Kernel-packages] [Bug 1942935] Re: kernel io hangs during mdcheck/resync
Here is Donald Buczek's reproducer script. I setup an Ubuntu 20.04 VM with latest linux-image-generic and was able to reproduce it within maybe 10 or 15 minutes. Exactly the same issue. Filesystem layout built as follows: # assemble raid devices mdadm --create /dev/md0 --level=1 --raid-devices=2 --spare-devices=1 /dev/sda2 /dev/sdb2 /dev/sdc2 mdadm --create /dev/md1 --level=5 --raid-devices=5 /dev/sda3 /dev/sdb3 /dev/sdc3 /dev/sdd3 /dev/sde3 # create PVs, VGs, LVs pvcreate /dev/md1 vgcreate vg1 /dev/md1 lvcreate --name root --extents 100%FREE vg1 # create filesystems mkfs.ext4 /dev/md0 mkfs.ext4 /dev/vg1/root ** Attachment added: "mdhang.sh" https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-5.11/+bug/1942935/+attachment/5525050/+files/mdhang.sh -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-signed-hwe-5.11 in Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync Status in linux-signed-hwe-5.11 package in Ubuntu: Confirmed Bug description: It seems to always occur during an mdcheck/resync, if I am logged in via SSH it is still somewhat responsive and basic utilities like dmesg will work. But it apppears any write I/O will hang the terminal and nothing is written to syslog (presumably because it is blocked). Below is output of dmesg and cat /proc/mdstat, it appears the data check was interrupted and /proc/mdstat still shows progress, and a whole slew of hung tasks including md1_resync itself. [756484.534293] md: data-check of RAID array md0 [756484.628039] md: delaying data-check of md1 until md0 has finished (they share one or more physical units) [756493.808773] md: md0: data-check done. [756493.829760] md: data-check of RAID array md1 [778112.446410] md: md1: data-check interrupted. [810654.608102] md: data-check of RAID array md1 [832291.201064] md: md1: data-check interrupted. [899745.389485] md: data-check of RAID array md1 [921395.835305] md: md1: data-check interrupted. [921588.558834] INFO: task systemd-journal:376 blocked for more than 120 seconds. [921588.558846] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.558850] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.558854] task:systemd-journal state:D stack:0 pid: 376 ppid: 1 flags:0x0220 [921588.558859] Call Trace: [921588.558864] __schedule+0x44c/0x8a0 [921588.558872] schedule+0x4f/0xc0 [921588.558876] md_write_start+0x150/0x240 [921588.558880] ? wait_woken+0x80/0x80 [921588.558886] raid5_make_request+0x88/0x890 [raid456] [921588.558898] ? wait_woken+0x80/0x80 [921588.558901] ? mempool_kmalloc+0x17/0x20 [921588.558904] md_handle_request+0x12d/0x1a0 [921588.558907] ? __part_start_io_acct+0x51/0xf0 [921588.558912] md_submit_bio+0xca/0x100 [921588.558915] submit_bio_noacct+0x112/0x4f0 [921588.558918] ? ext4_fc_reserve_space+0x110/0x230 [921588.558922] submit_bio+0x51/0x1a0 [921588.558925] ? _cond_resched+0x19/0x30 [921588.558928] ? kmem_cache_alloc+0x38e/0x440 [921588.558932] ? ext4_init_io_end+0x1f/0x50 [921588.558936] ext4_io_submit+0x4d/0x60 [921588.558940] ext4_writepages+0x2c6/0xcd0 [921588.558944] do_writepages+0x43/0xd0 [921588.558948] ? do_writepages+0x43/0xd0 [921588.558951] ? fault_dirty_shared_page+0xa5/0x110 [921588.558955] __filemap_fdatawrite_range+0xcc/0x110 [921588.558960] file_write_and_wait_range+0x74/0xc0 [921588.558962] ext4_sync_file+0xf5/0x350 [921588.558967] vfs_fsync_range+0x49/0x80 [921588.558970] do_fsync+0x3d/0x70 [921588.558973] __x64_sys_fsync+0x14/0x20 [921588.558976] do_syscall_64+0x38/0x90 [921588.558980] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921588.558984] RIP: 0033:0x7f4c97ee832b [921588.558987] RSP: 002b:7ffdceb29e50 EFLAGS: 0293 ORIG_RAX: 004a [921588.558991] RAX: ffda RBX: 55ced34b0fa0 RCX: 7f4c97ee832b [921588.558993] RDX: 7f4c97fc8000 RSI: 55ced3487b70 RDI: 0021 [921588.558995] RBP: 0001 R08: R09: 7ffdceb29fa8 [921588.558996] R10: 7f4c97d2c848 R11: 0293 R12: 7ffdceb29fa8 [921588.558998] R13: 7ffdceb29fa0 R14: 55ced34b0fa0 R15: 55ced34bcf90 [921588.559014] INFO: task mysqld:1505 blocked for more than 120 seconds. [921588.559018] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.559022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.559025] task:mysqld state:D stack:0 pid: 1505 ppid: 1 flags:0x [921588.559030] Call Trace: [921588.559032] __schedule+0x44c/0x8a0 [921588.559036] schedule+0x4f/0xc0 [921588.559040] md_write_start+0x150/0x240 [921588.559044] ? wait_woken+0x80/0x80 [921588.559047] raid5_make_request+0x88/0x890 [raid456]
[Kernel-packages] [Bug 1942935] Re: kernel io hangs during mdcheck/resync
The patch hasn't made it into mainline from what I have seen, it looks like it died back in March waiting for feedback from additional kernel developers. From what I have gathered this is a deadlock scenario directly caused by pausing the resync while the system is under heavy write activity. Donald Buczek provided a reproducer which is a shell script that generates a lot of write activity and pauses/resumes the raid scrubbing. And he also provided a workaround to get the stuck system running without reboot: echo active > /sys/block/md1/md/array_state I haven't tried the patch or any of this, I pretty much eliminated the trigger which is mdcheck_start & mdcheck_continue and went back to the 18.04 LTS way of scrubbing arrays (which is basically don't pause/interrupt it once it starts). I ran through a checkarray yesterday to 100%, no problems. Meanwhile since upgrading to 20.04 LTS it has hung almost every single time through 5.4, 5.8, and 5.11 kernels. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-signed-hwe-5.11 in Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync Status in linux-signed-hwe-5.11 package in Ubuntu: Confirmed Bug description: It seems to always occur during an mdcheck/resync, if I am logged in via SSH it is still somewhat responsive and basic utilities like dmesg will work. But it apppears any write I/O will hang the terminal and nothing is written to syslog (presumably because it is blocked). Below is output of dmesg and cat /proc/mdstat, it appears the data check was interrupted and /proc/mdstat still shows progress, and a whole slew of hung tasks including md1_resync itself. [756484.534293] md: data-check of RAID array md0 [756484.628039] md: delaying data-check of md1 until md0 has finished (they share one or more physical units) [756493.808773] md: md0: data-check done. [756493.829760] md: data-check of RAID array md1 [778112.446410] md: md1: data-check interrupted. [810654.608102] md: data-check of RAID array md1 [832291.201064] md: md1: data-check interrupted. [899745.389485] md: data-check of RAID array md1 [921395.835305] md: md1: data-check interrupted. [921588.558834] INFO: task systemd-journal:376 blocked for more than 120 seconds. [921588.558846] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.558850] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.558854] task:systemd-journal state:D stack:0 pid: 376 ppid: 1 flags:0x0220 [921588.558859] Call Trace: [921588.558864] __schedule+0x44c/0x8a0 [921588.558872] schedule+0x4f/0xc0 [921588.558876] md_write_start+0x150/0x240 [921588.558880] ? wait_woken+0x80/0x80 [921588.558886] raid5_make_request+0x88/0x890 [raid456] [921588.558898] ? wait_woken+0x80/0x80 [921588.558901] ? mempool_kmalloc+0x17/0x20 [921588.558904] md_handle_request+0x12d/0x1a0 [921588.558907] ? __part_start_io_acct+0x51/0xf0 [921588.558912] md_submit_bio+0xca/0x100 [921588.558915] submit_bio_noacct+0x112/0x4f0 [921588.558918] ? ext4_fc_reserve_space+0x110/0x230 [921588.558922] submit_bio+0x51/0x1a0 [921588.558925] ? _cond_resched+0x19/0x30 [921588.558928] ? kmem_cache_alloc+0x38e/0x440 [921588.558932] ? ext4_init_io_end+0x1f/0x50 [921588.558936] ext4_io_submit+0x4d/0x60 [921588.558940] ext4_writepages+0x2c6/0xcd0 [921588.558944] do_writepages+0x43/0xd0 [921588.558948] ? do_writepages+0x43/0xd0 [921588.558951] ? fault_dirty_shared_page+0xa5/0x110 [921588.558955] __filemap_fdatawrite_range+0xcc/0x110 [921588.558960] file_write_and_wait_range+0x74/0xc0 [921588.558962] ext4_sync_file+0xf5/0x350 [921588.558967] vfs_fsync_range+0x49/0x80 [921588.558970] do_fsync+0x3d/0x70 [921588.558973] __x64_sys_fsync+0x14/0x20 [921588.558976] do_syscall_64+0x38/0x90 [921588.558980] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921588.558984] RIP: 0033:0x7f4c97ee832b [921588.558987] RSP: 002b:7ffdceb29e50 EFLAGS: 0293 ORIG_RAX: 004a [921588.558991] RAX: ffda RBX: 55ced34b0fa0 RCX: 7f4c97ee832b [921588.558993] RDX: 7f4c97fc8000 RSI: 55ced3487b70 RDI: 0021 [921588.558995] RBP: 0001 R08: R09: 7ffdceb29fa8 [921588.558996] R10: 7f4c97d2c848 R11: 0293 R12: 7ffdceb29fa8 [921588.558998] R13: 7ffdceb29fa0 R14: 55ced34b0fa0 R15: 55ced34bcf90 [921588.559014] INFO: task mysqld:1505 blocked for more than 120 seconds. [921588.559018] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.559022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.559025] task:mysqld state:D stack:0 pid: 1505 ppid: 1 flags:0x [921588.559030] Call Trace: [921588.559032]
[Kernel-packages] [Bug 1942935] Re: kernel io hangs during mdcheck/resync
Here is the proposed patch, Doesn't appear to have been applied. Last report was with 5.11rc5. https://lore.kernel.org/linux-raid/1613177399-22024-1-git-send-email- guoqing.ji...@cloud.ionos.com/ -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-signed-hwe-5.11 in Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync Status in linux-signed-hwe-5.11 package in Ubuntu: New Bug description: It seems to always occur during an mdcheck/resync, if I am logged in via SSH it is still somewhat responsive and basic utilities like dmesg will work. But it apppears any write I/O will hang the terminal and nothing is written to syslog (presumably because it is blocked). Below is output of dmesg and cat /proc/mdstat, it appears the data check was interrupted and /proc/mdstat still shows progress, and a whole slew of hung tasks including md1_resync itself. [756484.534293] md: data-check of RAID array md0 [756484.628039] md: delaying data-check of md1 until md0 has finished (they share one or more physical units) [756493.808773] md: md0: data-check done. [756493.829760] md: data-check of RAID array md1 [778112.446410] md: md1: data-check interrupted. [810654.608102] md: data-check of RAID array md1 [832291.201064] md: md1: data-check interrupted. [899745.389485] md: data-check of RAID array md1 [921395.835305] md: md1: data-check interrupted. [921588.558834] INFO: task systemd-journal:376 blocked for more than 120 seconds. [921588.558846] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.558850] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.558854] task:systemd-journal state:D stack:0 pid: 376 ppid: 1 flags:0x0220 [921588.558859] Call Trace: [921588.558864] __schedule+0x44c/0x8a0 [921588.558872] schedule+0x4f/0xc0 [921588.558876] md_write_start+0x150/0x240 [921588.558880] ? wait_woken+0x80/0x80 [921588.558886] raid5_make_request+0x88/0x890 [raid456] [921588.558898] ? wait_woken+0x80/0x80 [921588.558901] ? mempool_kmalloc+0x17/0x20 [921588.558904] md_handle_request+0x12d/0x1a0 [921588.558907] ? __part_start_io_acct+0x51/0xf0 [921588.558912] md_submit_bio+0xca/0x100 [921588.558915] submit_bio_noacct+0x112/0x4f0 [921588.558918] ? ext4_fc_reserve_space+0x110/0x230 [921588.558922] submit_bio+0x51/0x1a0 [921588.558925] ? _cond_resched+0x19/0x30 [921588.558928] ? kmem_cache_alloc+0x38e/0x440 [921588.558932] ? ext4_init_io_end+0x1f/0x50 [921588.558936] ext4_io_submit+0x4d/0x60 [921588.558940] ext4_writepages+0x2c6/0xcd0 [921588.558944] do_writepages+0x43/0xd0 [921588.558948] ? do_writepages+0x43/0xd0 [921588.558951] ? fault_dirty_shared_page+0xa5/0x110 [921588.558955] __filemap_fdatawrite_range+0xcc/0x110 [921588.558960] file_write_and_wait_range+0x74/0xc0 [921588.558962] ext4_sync_file+0xf5/0x350 [921588.558967] vfs_fsync_range+0x49/0x80 [921588.558970] do_fsync+0x3d/0x70 [921588.558973] __x64_sys_fsync+0x14/0x20 [921588.558976] do_syscall_64+0x38/0x90 [921588.558980] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921588.558984] RIP: 0033:0x7f4c97ee832b [921588.558987] RSP: 002b:7ffdceb29e50 EFLAGS: 0293 ORIG_RAX: 004a [921588.558991] RAX: ffda RBX: 55ced34b0fa0 RCX: 7f4c97ee832b [921588.558993] RDX: 7f4c97fc8000 RSI: 55ced3487b70 RDI: 0021 [921588.558995] RBP: 0001 R08: R09: 7ffdceb29fa8 [921588.558996] R10: 7f4c97d2c848 R11: 0293 R12: 7ffdceb29fa8 [921588.558998] R13: 7ffdceb29fa0 R14: 55ced34b0fa0 R15: 55ced34bcf90 [921588.559014] INFO: task mysqld:1505 blocked for more than 120 seconds. [921588.559018] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.559022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.559025] task:mysqld state:D stack:0 pid: 1505 ppid: 1 flags:0x [921588.559030] Call Trace: [921588.559032] __schedule+0x44c/0x8a0 [921588.559036] schedule+0x4f/0xc0 [921588.559040] md_write_start+0x150/0x240 [921588.559044] ? wait_woken+0x80/0x80 [921588.559047] raid5_make_request+0x88/0x890 [raid456] [921588.559056] ? wait_woken+0x80/0x80 [921588.559059] ? mempool_kmalloc+0x17/0x20 [921588.559062] md_handle_request+0x12d/0x1a0 [921588.559065] ? __part_start_io_acct+0x51/0xf0 [921588.559068] md_submit_bio+0xca/0x100 [921588.559071] submit_bio_noacct+0x112/0x4f0 [921588.559075] submit_bio+0x51/0x1a0 [921588.559077] ? _cond_resched+0x19/0x30 [921588.559081] ? kmem_cache_alloc+0x38e/0x440 [921588.559084] ? ext4_init_io_end+0x1f/0x50 [921588.559088] ext4_io_submit+0x4d/0x60 [921588.559091] ext4_writepages+0x2c6/0xcd0 [921588.559094] ?
[Kernel-packages] [Bug 1942935] Re: kernel io hangs during mdcheck/resync
Similar report here on 5.10.0-rc4: https://www.spinics.net/lists/raid/msg66654.html I ended up masking the services introduced with 20.04 LTS, and switched back the crontab. systemctl mask mdcheck_continue.service mdcheck_continue.timer mdcheck_start.service mdcheck_start.timer cat > /etc/cron.d/mdadm << 'EOF' # # cron.d/mdadm -- schedules periodic redundancy checks of MD devices # # Copyright © martin f. krafft # distributed under the terms of the Artistic Licence 2.0 # # By default, run at 00:57 on every Sunday, but do nothing unless the day of # the month is less than or equal to 7. Thus, only run on the first Sunday of # each month. crontab(5) sucks, unfortunately, in this regard; therefore this # hack (see #380425). 57 0 * * 0 root if [ -x /usr/share/mdadm/checkarray ] && [ $(date +\%d) -le 7 ]; then /usr/share/mdadm/checkarray --cron --all --idle --quiet; fi EOF The pausing and resuming of the integrity check was an annoyance for me anyways. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-signed-hwe-5.11 in Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync Status in linux-signed-hwe-5.11 package in Ubuntu: New Bug description: It seems to always occur during an mdcheck/resync, if I am logged in via SSH it is still somewhat responsive and basic utilities like dmesg will work. But it apppears any write I/O will hang the terminal and nothing is written to syslog (presumably because it is blocked). Below is output of dmesg and cat /proc/mdstat, it appears the data check was interrupted and /proc/mdstat still shows progress, and a whole slew of hung tasks including md1_resync itself. [756484.534293] md: data-check of RAID array md0 [756484.628039] md: delaying data-check of md1 until md0 has finished (they share one or more physical units) [756493.808773] md: md0: data-check done. [756493.829760] md: data-check of RAID array md1 [778112.446410] md: md1: data-check interrupted. [810654.608102] md: data-check of RAID array md1 [832291.201064] md: md1: data-check interrupted. [899745.389485] md: data-check of RAID array md1 [921395.835305] md: md1: data-check interrupted. [921588.558834] INFO: task systemd-journal:376 blocked for more than 120 seconds. [921588.558846] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.558850] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.558854] task:systemd-journal state:D stack:0 pid: 376 ppid: 1 flags:0x0220 [921588.558859] Call Trace: [921588.558864] __schedule+0x44c/0x8a0 [921588.558872] schedule+0x4f/0xc0 [921588.558876] md_write_start+0x150/0x240 [921588.558880] ? wait_woken+0x80/0x80 [921588.558886] raid5_make_request+0x88/0x890 [raid456] [921588.558898] ? wait_woken+0x80/0x80 [921588.558901] ? mempool_kmalloc+0x17/0x20 [921588.558904] md_handle_request+0x12d/0x1a0 [921588.558907] ? __part_start_io_acct+0x51/0xf0 [921588.558912] md_submit_bio+0xca/0x100 [921588.558915] submit_bio_noacct+0x112/0x4f0 [921588.558918] ? ext4_fc_reserve_space+0x110/0x230 [921588.558922] submit_bio+0x51/0x1a0 [921588.558925] ? _cond_resched+0x19/0x30 [921588.558928] ? kmem_cache_alloc+0x38e/0x440 [921588.558932] ? ext4_init_io_end+0x1f/0x50 [921588.558936] ext4_io_submit+0x4d/0x60 [921588.558940] ext4_writepages+0x2c6/0xcd0 [921588.558944] do_writepages+0x43/0xd0 [921588.558948] ? do_writepages+0x43/0xd0 [921588.558951] ? fault_dirty_shared_page+0xa5/0x110 [921588.558955] __filemap_fdatawrite_range+0xcc/0x110 [921588.558960] file_write_and_wait_range+0x74/0xc0 [921588.558962] ext4_sync_file+0xf5/0x350 [921588.558967] vfs_fsync_range+0x49/0x80 [921588.558970] do_fsync+0x3d/0x70 [921588.558973] __x64_sys_fsync+0x14/0x20 [921588.558976] do_syscall_64+0x38/0x90 [921588.558980] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921588.558984] RIP: 0033:0x7f4c97ee832b [921588.558987] RSP: 002b:7ffdceb29e50 EFLAGS: 0293 ORIG_RAX: 004a [921588.558991] RAX: ffda RBX: 55ced34b0fa0 RCX: 7f4c97ee832b [921588.558993] RDX: 7f4c97fc8000 RSI: 55ced3487b70 RDI: 0021 [921588.558995] RBP: 0001 R08: R09: 7ffdceb29fa8 [921588.558996] R10: 7f4c97d2c848 R11: 0293 R12: 7ffdceb29fa8 [921588.558998] R13: 7ffdceb29fa0 R14: 55ced34b0fa0 R15: 55ced34bcf90 [921588.559014] INFO: task mysqld:1505 blocked for more than 120 seconds. [921588.559018] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.559022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.559025] task:mysqld state:D stack:0 pid: 1505 ppid: 1 flags:0x [921588.559030] Call Trace: [921588.559032] __schedule+0x44c/0x8a0
[Kernel-packages] [Bug 1942935] Re: kernel io hangs during mdcheck/resync
Hi Kleber, I installed it later yesterday, but I won't know until the next resync. This has been a problem since at least linux 5.4 kernel that shipped with Ubuntu 20.04. I don't think I had these problems on Ubuntu 18.04 LTS, the same hardware, running the linux-image-generic at that time. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-signed-hwe-5.11 in Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync Status in linux-signed-hwe-5.11 package in Ubuntu: New Bug description: It seems to always occur during an mdcheck/resync, if I am logged in via SSH it is still somewhat responsive and basic utilities like dmesg will work. But it apppears any write I/O will hang the terminal and nothing is written to syslog (presumably because it is blocked). Below is output of dmesg and cat /proc/mdstat, it appears the data check was interrupted and /proc/mdstat still shows progress, and a whole slew of hung tasks including md1_resync itself. [756484.534293] md: data-check of RAID array md0 [756484.628039] md: delaying data-check of md1 until md0 has finished (they share one or more physical units) [756493.808773] md: md0: data-check done. [756493.829760] md: data-check of RAID array md1 [778112.446410] md: md1: data-check interrupted. [810654.608102] md: data-check of RAID array md1 [832291.201064] md: md1: data-check interrupted. [899745.389485] md: data-check of RAID array md1 [921395.835305] md: md1: data-check interrupted. [921588.558834] INFO: task systemd-journal:376 blocked for more than 120 seconds. [921588.558846] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.558850] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.558854] task:systemd-journal state:D stack:0 pid: 376 ppid: 1 flags:0x0220 [921588.558859] Call Trace: [921588.558864] __schedule+0x44c/0x8a0 [921588.558872] schedule+0x4f/0xc0 [921588.558876] md_write_start+0x150/0x240 [921588.558880] ? wait_woken+0x80/0x80 [921588.558886] raid5_make_request+0x88/0x890 [raid456] [921588.558898] ? wait_woken+0x80/0x80 [921588.558901] ? mempool_kmalloc+0x17/0x20 [921588.558904] md_handle_request+0x12d/0x1a0 [921588.558907] ? __part_start_io_acct+0x51/0xf0 [921588.558912] md_submit_bio+0xca/0x100 [921588.558915] submit_bio_noacct+0x112/0x4f0 [921588.558918] ? ext4_fc_reserve_space+0x110/0x230 [921588.558922] submit_bio+0x51/0x1a0 [921588.558925] ? _cond_resched+0x19/0x30 [921588.558928] ? kmem_cache_alloc+0x38e/0x440 [921588.558932] ? ext4_init_io_end+0x1f/0x50 [921588.558936] ext4_io_submit+0x4d/0x60 [921588.558940] ext4_writepages+0x2c6/0xcd0 [921588.558944] do_writepages+0x43/0xd0 [921588.558948] ? do_writepages+0x43/0xd0 [921588.558951] ? fault_dirty_shared_page+0xa5/0x110 [921588.558955] __filemap_fdatawrite_range+0xcc/0x110 [921588.558960] file_write_and_wait_range+0x74/0xc0 [921588.558962] ext4_sync_file+0xf5/0x350 [921588.558967] vfs_fsync_range+0x49/0x80 [921588.558970] do_fsync+0x3d/0x70 [921588.558973] __x64_sys_fsync+0x14/0x20 [921588.558976] do_syscall_64+0x38/0x90 [921588.558980] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921588.558984] RIP: 0033:0x7f4c97ee832b [921588.558987] RSP: 002b:7ffdceb29e50 EFLAGS: 0293 ORIG_RAX: 004a [921588.558991] RAX: ffda RBX: 55ced34b0fa0 RCX: 7f4c97ee832b [921588.558993] RDX: 7f4c97fc8000 RSI: 55ced3487b70 RDI: 0021 [921588.558995] RBP: 0001 R08: R09: 7ffdceb29fa8 [921588.558996] R10: 7f4c97d2c848 R11: 0293 R12: 7ffdceb29fa8 [921588.558998] R13: 7ffdceb29fa0 R14: 55ced34b0fa0 R15: 55ced34bcf90 [921588.559014] INFO: task mysqld:1505 blocked for more than 120 seconds. [921588.559018] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.559022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.559025] task:mysqld state:D stack:0 pid: 1505 ppid: 1 flags:0x [921588.559030] Call Trace: [921588.559032] __schedule+0x44c/0x8a0 [921588.559036] schedule+0x4f/0xc0 [921588.559040] md_write_start+0x150/0x240 [921588.559044] ? wait_woken+0x80/0x80 [921588.559047] raid5_make_request+0x88/0x890 [raid456] [921588.559056] ? wait_woken+0x80/0x80 [921588.559059] ? mempool_kmalloc+0x17/0x20 [921588.559062] md_handle_request+0x12d/0x1a0 [921588.559065] ? __part_start_io_acct+0x51/0xf0 [921588.559068] md_submit_bio+0xca/0x100 [921588.559071] submit_bio_noacct+0x112/0x4f0 [921588.559075] submit_bio+0x51/0x1a0 [921588.559077] ? _cond_resched+0x19/0x30 [921588.559081] ? kmem_cache_alloc+0x38e/0x440 [921588.559084] ? ext4_init_io_end+0x1f/0x50 [921588.559088]
[Kernel-packages] [Bug 1942935] Re: kernel io hangs during mdcheck/resync
** Attachment added: "screenlog.txt" https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-5.11/+bug/1942935/+attachment/5523575/+files/screenlog.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-signed-hwe-5.11 in Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync Status in linux-signed-hwe-5.11 package in Ubuntu: New Bug description: It seems to always occur during an mdcheck/resync, if I am logged in via SSH it is still somewhat responsive and basic utilities like dmesg will work. But it apppears any write I/O will hang the terminal and nothing is written to syslog (presumably because it is blocked). Below is output of dmesg and cat /proc/mdstat, it appears the data check was interrupted and /proc/mdstat still shows progress, and a whole slew of hung tasks including md1_resync itself. [756484.534293] md: data-check of RAID array md0 [756484.628039] md: delaying data-check of md1 until md0 has finished (they share one or more physical units) [756493.808773] md: md0: data-check done. [756493.829760] md: data-check of RAID array md1 [778112.446410] md: md1: data-check interrupted. [810654.608102] md: data-check of RAID array md1 [832291.201064] md: md1: data-check interrupted. [899745.389485] md: data-check of RAID array md1 [921395.835305] md: md1: data-check interrupted. [921588.558834] INFO: task systemd-journal:376 blocked for more than 120 seconds. [921588.558846] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.558850] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.558854] task:systemd-journal state:D stack:0 pid: 376 ppid: 1 flags:0x0220 [921588.558859] Call Trace: [921588.558864] __schedule+0x44c/0x8a0 [921588.558872] schedule+0x4f/0xc0 [921588.558876] md_write_start+0x150/0x240 [921588.558880] ? wait_woken+0x80/0x80 [921588.558886] raid5_make_request+0x88/0x890 [raid456] [921588.558898] ? wait_woken+0x80/0x80 [921588.558901] ? mempool_kmalloc+0x17/0x20 [921588.558904] md_handle_request+0x12d/0x1a0 [921588.558907] ? __part_start_io_acct+0x51/0xf0 [921588.558912] md_submit_bio+0xca/0x100 [921588.558915] submit_bio_noacct+0x112/0x4f0 [921588.558918] ? ext4_fc_reserve_space+0x110/0x230 [921588.558922] submit_bio+0x51/0x1a0 [921588.558925] ? _cond_resched+0x19/0x30 [921588.558928] ? kmem_cache_alloc+0x38e/0x440 [921588.558932] ? ext4_init_io_end+0x1f/0x50 [921588.558936] ext4_io_submit+0x4d/0x60 [921588.558940] ext4_writepages+0x2c6/0xcd0 [921588.558944] do_writepages+0x43/0xd0 [921588.558948] ? do_writepages+0x43/0xd0 [921588.558951] ? fault_dirty_shared_page+0xa5/0x110 [921588.558955] __filemap_fdatawrite_range+0xcc/0x110 [921588.558960] file_write_and_wait_range+0x74/0xc0 [921588.558962] ext4_sync_file+0xf5/0x350 [921588.558967] vfs_fsync_range+0x49/0x80 [921588.558970] do_fsync+0x3d/0x70 [921588.558973] __x64_sys_fsync+0x14/0x20 [921588.558976] do_syscall_64+0x38/0x90 [921588.558980] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921588.558984] RIP: 0033:0x7f4c97ee832b [921588.558987] RSP: 002b:7ffdceb29e50 EFLAGS: 0293 ORIG_RAX: 004a [921588.558991] RAX: ffda RBX: 55ced34b0fa0 RCX: 7f4c97ee832b [921588.558993] RDX: 7f4c97fc8000 RSI: 55ced3487b70 RDI: 0021 [921588.558995] RBP: 0001 R08: R09: 7ffdceb29fa8 [921588.558996] R10: 7f4c97d2c848 R11: 0293 R12: 7ffdceb29fa8 [921588.558998] R13: 7ffdceb29fa0 R14: 55ced34b0fa0 R15: 55ced34bcf90 [921588.559014] INFO: task mysqld:1505 blocked for more than 120 seconds. [921588.559018] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.559022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.559025] task:mysqld state:D stack:0 pid: 1505 ppid: 1 flags:0x [921588.559030] Call Trace: [921588.559032] __schedule+0x44c/0x8a0 [921588.559036] schedule+0x4f/0xc0 [921588.559040] md_write_start+0x150/0x240 [921588.559044] ? wait_woken+0x80/0x80 [921588.559047] raid5_make_request+0x88/0x890 [raid456] [921588.559056] ? wait_woken+0x80/0x80 [921588.559059] ? mempool_kmalloc+0x17/0x20 [921588.559062] md_handle_request+0x12d/0x1a0 [921588.559065] ? __part_start_io_acct+0x51/0xf0 [921588.559068] md_submit_bio+0xca/0x100 [921588.559071] submit_bio_noacct+0x112/0x4f0 [921588.559075] submit_bio+0x51/0x1a0 [921588.559077] ? _cond_resched+0x19/0x30 [921588.559081] ? kmem_cache_alloc+0x38e/0x440 [921588.559084] ? ext4_init_io_end+0x1f/0x50 [921588.559088] ext4_io_submit+0x4d/0x60 [921588.559091] ext4_writepages+0x2c6/0xcd0 [921588.559094] ? __schedule+0x454/0x8a0 [921588.559097] ?
[Kernel-packages] [Bug 1942935] [NEW] kernel io hangs during mdcheck/resync
Public bug reported: It seems to always occur during an mdcheck/resync, if I am logged in via SSH it is still somewhat responsive and basic utilities like dmesg will work. But it apppears any write I/O will hang the terminal and nothing is written to syslog (presumably because it is blocked). Below is output of dmesg and cat /proc/mdstat, it appears the data check was interrupted and /proc/mdstat still shows progress, and a whole slew of hung tasks including md1_resync itself. [756484.534293] md: data-check of RAID array md0 [756484.628039] md: delaying data-check of md1 until md0 has finished (they share one or more physical units) [756493.808773] md: md0: data-check done. [756493.829760] md: data-check of RAID array md1 [778112.446410] md: md1: data-check interrupted. [810654.608102] md: data-check of RAID array md1 [832291.201064] md: md1: data-check interrupted. [899745.389485] md: data-check of RAID array md1 [921395.835305] md: md1: data-check interrupted. [921588.558834] INFO: task systemd-journal:376 blocked for more than 120 seconds. [921588.558846] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.558850] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.558854] task:systemd-journal state:D stack:0 pid: 376 ppid: 1 flags:0x0220 [921588.558859] Call Trace: [921588.558864] __schedule+0x44c/0x8a0 [921588.558872] schedule+0x4f/0xc0 [921588.558876] md_write_start+0x150/0x240 [921588.558880] ? wait_woken+0x80/0x80 [921588.558886] raid5_make_request+0x88/0x890 [raid456] [921588.558898] ? wait_woken+0x80/0x80 [921588.558901] ? mempool_kmalloc+0x17/0x20 [921588.558904] md_handle_request+0x12d/0x1a0 [921588.558907] ? __part_start_io_acct+0x51/0xf0 [921588.558912] md_submit_bio+0xca/0x100 [921588.558915] submit_bio_noacct+0x112/0x4f0 [921588.558918] ? ext4_fc_reserve_space+0x110/0x230 [921588.558922] submit_bio+0x51/0x1a0 [921588.558925] ? _cond_resched+0x19/0x30 [921588.558928] ? kmem_cache_alloc+0x38e/0x440 [921588.558932] ? ext4_init_io_end+0x1f/0x50 [921588.558936] ext4_io_submit+0x4d/0x60 [921588.558940] ext4_writepages+0x2c6/0xcd0 [921588.558944] do_writepages+0x43/0xd0 [921588.558948] ? do_writepages+0x43/0xd0 [921588.558951] ? fault_dirty_shared_page+0xa5/0x110 [921588.558955] __filemap_fdatawrite_range+0xcc/0x110 [921588.558960] file_write_and_wait_range+0x74/0xc0 [921588.558962] ext4_sync_file+0xf5/0x350 [921588.558967] vfs_fsync_range+0x49/0x80 [921588.558970] do_fsync+0x3d/0x70 [921588.558973] __x64_sys_fsync+0x14/0x20 [921588.558976] do_syscall_64+0x38/0x90 [921588.558980] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921588.558984] RIP: 0033:0x7f4c97ee832b [921588.558987] RSP: 002b:7ffdceb29e50 EFLAGS: 0293 ORIG_RAX: 004a [921588.558991] RAX: ffda RBX: 55ced34b0fa0 RCX: 7f4c97ee832b [921588.558993] RDX: 7f4c97fc8000 RSI: 55ced3487b70 RDI: 0021 [921588.558995] RBP: 0001 R08: R09: 7ffdceb29fa8 [921588.558996] R10: 7f4c97d2c848 R11: 0293 R12: 7ffdceb29fa8 [921588.558998] R13: 7ffdceb29fa0 R14: 55ced34b0fa0 R15: 55ced34bcf90 [921588.559014] INFO: task mysqld:1505 blocked for more than 120 seconds. [921588.559018] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.559022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.559025] task:mysqld state:D stack:0 pid: 1505 ppid: 1 flags:0x [921588.559030] Call Trace: [921588.559032] __schedule+0x44c/0x8a0 [921588.559036] schedule+0x4f/0xc0 [921588.559040] md_write_start+0x150/0x240 [921588.559044] ? wait_woken+0x80/0x80 [921588.559047] raid5_make_request+0x88/0x890 [raid456] [921588.559056] ? wait_woken+0x80/0x80 [921588.559059] ? mempool_kmalloc+0x17/0x20 [921588.559062] md_handle_request+0x12d/0x1a0 [921588.559065] ? __part_start_io_acct+0x51/0xf0 [921588.559068] md_submit_bio+0xca/0x100 [921588.559071] submit_bio_noacct+0x112/0x4f0 [921588.559075] submit_bio+0x51/0x1a0 [921588.559077] ? _cond_resched+0x19/0x30 [921588.559081] ? kmem_cache_alloc+0x38e/0x440 [921588.559084] ? ext4_init_io_end+0x1f/0x50 [921588.559088] ext4_io_submit+0x4d/0x60 [921588.559091] ext4_writepages+0x2c6/0xcd0 [921588.559094] ? __schedule+0x454/0x8a0 [921588.559097] ? hrtimer_start_range_ns+0x1aa/0x2f0 [921588.559100] ? timerqueue_del+0x24/0x50 [921588.559105] ? futex_wait+0x1ed/0x270 [921588.559109] do_writepages+0x43/0xd0 [921588.559112] ? do_writepages+0x43/0xd0 [921588.559115] ? futex_wake+0x7c/0x180 [921588.559118] __filemap_fdatawrite_range+0xcc/0x110 [921588.559123] file_write_and_wait_range+0x74/0xc0 [921588.559126] ext4_sync_file+0xf5/0x350 [921588.559129] vfs_fsync_range+0x49/0x80 [921588.559132] ? __fget_light+0x62/0x80 [921588.559136] do_fsync+0x3d/0x70 [921588.559139] __x64_sys_fsync+0x14/0x20 [921588.559142] do_syscall_64+0x38/0x90
