I just got bit by this, too. Was transferring ~40T via rsync to a new NAS setup... got about 12T in and it just stopped. Eventually realized the mountpoint was unresponsive. Rebooting and kicking rsync back off resulted in the same scenario almost immediately. The machine in question is running 5.15.27. I'm going to upgrade the kernel and see if the problem persists... Does anyone know if this is fixed or being worked on upstream already?
-- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1972898 Title: Kernel Bug: 22.04,EXT4, samba (smbd) on MDADM raid6: Copying large volume of files. Status in linux package in Ubuntu: Confirmed Bug description: 60 Drive MDADM Raid 6, ext4, Ubuntu 22.04. Issue reproduced on both Supermicro SSG-6048R and HP ProLiant DL380 servers. System was stable on Ubuntu 20.04. Unstable following upgrade to 22.04 (kernel version 5.15) To reproduce kernel error, copy thousands of files (~1tb of data) to samba-share from any windows computer. After some time(seconds to minutes), a Kernel error is thrown, smbd process is unresponsive and cannot be killed, file transfer stops, the mounted drive freezes (directory operations including ls,mv,cp on the mount are not possible) and the system needs to be hard-rebooted. Quite an unhappy outcome :) I then moved the 60 drives to an external enclosure, and connected to a new computer (HP ProLiant DL380). After assembling the raid drive, with a fresh install of Ubuntu 22.04 on the new system, the kernel error was reproduced. I cannot reproduce the error copying via nfs or copying files on the drive itself. Single files or small transfers proceed without error. Filesystem passes fsck. Happy to assist in troubleshooting in any way. Kernel error message from both systems follows. **********New System (HP ProLiant DL380) Kernel Error************** May 10 01:32:49 nas3 kernel: [ 1463.900175] ------------[ cut here ]------------ May 10 01:32:49 nas3 kernel: [ 1463.900179] kernel BUG at fs/ext4/xattr.c:2071! May 10 01:32:49 nas3 kernel: [ 1463.900214] invalid opcode: 0000 [#1] SMP PTI May 10 01:32:49 nas3 kernel: [ 1463.900233] CPU: 0 PID: 5989 Comm: smbd Not tainted 5.15.0-27-generic #28-Ubuntu May 10 01:32:49 nas3 kernel: [ 1463.900939] Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 04/25/2017 May 10 01:32:49 nas3 kernel: [ 1463.901560] RIP: 0010:ext4_xattr_block_set+0xbba/0xbd0 May 10 01:32:49 nas3 kernel: [ 1463.902190] Code: c7 45 8c f4 ff ff ff eb b4 48 8b 7d 90 48 c7 c1 7f 12 61 a8 ba 2d 08 00 00 48 c7 c6 d0 3c 25 a8 e8 9b 6f ff ff e9 a5 fe ff ff <0> May 10 01:32:49 nas3 kernel: [ 1463.903445] RSP: 0018:ffffa59e0b51f9c0 EFLAGS: 00010206 May 10 01:32:49 nas3 kernel: [ 1463.904080] RAX: 0000000000000003 RBX: ffff97aa0490b680 RCX: ffffffffa860a8e7 May 10 01:32:49 nas3 kernel: [ 1463.904727] RDX: 0000000000000261 RSI: 0000000000000000 RDI: 000000000003cca0 May 10 01:32:49 nas3 kernel: [ 1463.905384] RBP: ffffa59e0b51fa70 R08: ffff97aa21824138 R09: 0000000000000000 May 10 01:32:49 nas3 kernel: [ 1463.906051] R10: ffff97aa0f6e87e0 R11: ffff97aae9073ff0 R12: 0000000000000000 May 10 01:32:49 nas3 kernel: [ 1463.906738] R13: ffff97ada77feac0 R14: 00000003165b0000 R15: 0000000000000000 May 10 01:32:49 nas3 kernel: [ 1463.907411] FS: 00007f06ceb61a40(0000) GS:ffff97b93f800000(0000) knlGS:0000000000000000 May 10 01:32:49 nas3 kernel: [ 1463.908049] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 10 01:32:49 nas3 kernel: [ 1463.908697] CR2: 000055c076d0d4f8 CR3: 000000029e6fe003 CR4: 00000000001706f0 May 10 01:32:49 nas3 kernel: [ 1463.909349] Call Trace: May 10 01:32:49 nas3 kernel: [ 1463.909989] <TASK> May 10 01:32:49 nas3 kernel: [ 1463.910624] ? jbd2_journal_get_write_access+0x43/0x90 May 10 01:32:49 nas3 kernel: [ 1463.911360] ext4_xattr_set_handle+0x487/0x620 May 10 01:32:49 nas3 kernel: [ 1463.912032] __ext4_set_acl+0xc1/0x130 May 10 01:32:49 nas3 kernel: [ 1463.912689] ext4_init_acl+0xe8/0x160 May 10 01:32:49 nas3 kernel: [ 1463.913327] __ext4_new_inode+0xf60/0x14e0 May 10 01:32:49 nas3 kernel: [ 1463.913962] ? path_parentat+0x4c/0x90 May 10 01:32:49 nas3 kernel: [ 1463.914595] ext4_mkdir+0x157/0x330 May 10 01:32:49 nas3 kernel: [ 1463.915265] vfs_mkdir+0x142/0x200 May 10 01:32:49 nas3 kernel: [ 1463.915883] do_mkdirat+0x120/0x140 May 10 01:32:49 nas3 kernel: [ 1463.916501] __x64_sys_mkdirat+0x51/0x70 May 10 01:32:49 nas3 kernel: [ 1463.917115] do_syscall_64+0x5c/0xc0 May 10 01:32:49 nas3 kernel: [ 1463.917733] ? exit_to_user_mode_prepare+0x37/0xb0 May 10 01:32:49 nas3 kernel: [ 1463.918365] ? syscall_exit_to_user_mode+0x27/0x50 May 10 01:32:49 nas3 kernel: [ 1463.919035] ? __x64_sys_newfstatat+0x1c/0x20 May 10 01:32:49 nas3 kernel: [ 1463.919665] ? do_syscall_64+0x69/0xc0 May 10 01:32:49 nas3 kernel: [ 1463.920300] ? __x64_sys_newfstatat+0x1c/0x20 May 10 01:32:49 nas3 kernel: [ 1463.920929] ? do_syscall_64+0x69/0xc0 May 10 01:32:49 nas3 kernel: [ 1463.921534] ? __x64_sys_newfstatat+0x1c/0x20 May 10 01:32:49 nas3 kernel: [ 1463.922121] ? do_syscall_64+0x69/0xc0 May 10 01:32:49 nas3 kernel: [ 1463.922703] ? syscall_exit_to_user_mode+0x27/0x50 May 10 01:32:49 nas3 kernel: [ 1463.923319] ? __x64_sys_readlink+0x1e/0x30 May 10 01:32:49 nas3 kernel: [ 1463.923893] ? do_syscall_64+0x69/0xc0 May 10 01:32:49 nas3 kernel: [ 1463.924467] ? do_syscall_64+0x69/0xc0 May 10 01:32:49 nas3 kernel: [ 1463.925025] entry_SYSCALL_64_after_hwframe+0x44/0xae May 10 01:32:49 nas3 kernel: [ 1463.925580] RIP: 0033:0x7f06d2dac63b **********Original System (*Supermicro SSG-6048R) * Kernel Error:**************** May 3 16:48:05 nas kernel: [31727.490899] ------------[ cut here ]------------ May 3 16:48:05 nas kernel: [31727.490906] kernel BUG at fs/ext4/xattr.c:2071! May 3 16:48:05 nas kernel: [31727.490920] invalid opcode: 0000 [#1] SMP PTI May 3 16:48:05 nas kernel: [31727.490927] CPU: 26 PID: 22593 Comm: smbd Not tainted 5.15.0-27-generic #28-Ubuntu May 3 16:48:05 nas kernel: [31727.490934] Hardware name: Supermicro SSG-6048R-E1CR60L/X10DSC+, BIOS 3.2 11/19/2019 May 3 16:48:05 nas kernel: [31727.490937] RIP: 0010:ext4_xattr_block_set+ 0xbba/0xbd0 May 3 16:48:05 nas kernel: [31727.490953] Code: c7 45 8c f4 ff ff ff eb b4 48 8b 7d 90 48 c7 c1 7f 12 e1 b7 ba 2d 08 00 00 48 c7 c6 d0 3c a5 b7 e8 9b 6f ff ff e9 a5 fe ff ff <0f> 0b e8 0f d8 89 00 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 May 3 16:48:05 nas kernel: [31727.490958] RSP: 0018:ffffbf548ba6fa50 EFLAGS: 00010203 May 3 16:48:05 nas kernel: [31727.490965] RAX: 0000000000000002 RBX: ffff9a914460ed80 RCX: ffffffffb7e0a8e7 May 3 16:48:05 nas kernel: [31727.490970] RDX: 0000000000000099 RSI: 0000000000000000 RDI: 000000000003cda0 May 3 16:48:05 nas kernel: [31727.490973] RBP: ffffbf548ba6fb00 R08: ffff9a9167c4cf08 R09: 0000000000000000 May 3 16:48:05 nas kernel: [31727.490977] R10: ffff9a81db1e1120 R11: ffff9a92046b8ff0 R12: 0000000000000000 May 3 16:48:05 nas kernel: [31727.490980] R13: ffff9a820b1dd3c8 R14: 00000002f3798000 R15: 0000000000000000 May 3 16:48:05 nas kernel: [31727.490984] FS: 00007f3491bc9a40(0000) GS:ffff9a913fb80000(0000) knlGS:0000000000000000 May 3 16:48:05 nas kernel: [31727.490988] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 3 16:48:05 nas kernel: [31727.490992] CR2: 000055573b9a2b78 CR3: 000000110ca1c002 CR4: 00000000003706e0 May 3 16:48:05 nas kernel: [31727.490997] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 May 3 16:48:05 nas kernel: [31727.491000] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 May 3 16:48:05 nas kernel: [31727.491003] Call Trace: May 3 16:48:05 nas kernel: [31727.491007] <TASK> May 3 16:48:05 nas kernel: [31727.491010] ? jbd2_journal_get_write_access+ 0x43/0x90 May 3 16:48:05 nas kernel: [31727.491022] ext4_xattr_set_handle+0x487/ 0x620 May 3 16:48:05 nas kernel: [31727.491032] __ext4_set_acl+0xc1/0x130 May 3 16:48:05 nas kernel: [31727.491038] ext4_init_acl+0xe8/0x160 May 3 16:48:05 nas kernel: [31727.491043] __ext4_new_inode+0xf60/0x14e0 May 3 16:48:05 nas kernel: [31727.491048] ? path_parentat+0x4c/0x90 May 3 16:48:05 nas kernel: [31727.491057] ? __find_next_entry+0x144/0x180 May 3 16:48:05 nas kernel: [31727.491067] ext4_mkdir+0x157/0x330 May 3 16:48:05 nas kernel: [31727.491075] vfs_mkdir+0x142/0x200 May 3 16:48:05 nas kernel: [31727.491097] do_mkdirat+0x120/0x140 May 3 16:48:05 nas kernel: [31727.491102] __x64_sys_mkdirat+0x51/0x70 May 3 16:48:05 nas kernel: [31727.491105] do_syscall_64+0x5c/0xc0 May 3 16:48:05 nas kernel: [31727.491113] ? exit_to_user_mode_prepare+ 0x37/0xb0 May 3 16:48:05 nas kernel: [31727.491120] ? syscall_exit_to_user_mode+ 0x27/0x50 May 3 16:48:05 nas kernel: [31727.491124] ? __x64_sys_getxattr+0x24/0x30 May 3 16:48:05 nas kernel: [31727.491132] ? do_syscall_64+0x69/0xc0 May 3 16:48:05 nas kernel: [31727.491137] entry_SYSCALL_64_after_ hwframe+0x44/0xae May 3 16:48:05 nas kernel: [31727.491143] RIP: 0033:0x7f3495e1463b May 3 16:48:05 nas kernel: [31727.491147] Code: 73 01 c3 48 8b 0d f5 47 10 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 02 01 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d c5 47 10 00 f7 d8 64 89 01 48 May 3 16:48:05 nas kernel: [31727.491151] RSP: 002b:00007ffe0f0e0b58 EFLAGS: 00000206 ORIG_RAX: 0000000000000102 May 3 16:48:05 nas kernel: [31727.491154] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f3495e1463b May 3 16:48:05 nas kernel: [31727.491157] RDX: 00000000000001ed RSI: 000055573b959e00 RDI: 0000000000000009 May 3 16:48:05 nas kernel: [31727.491159] RBP: 000055573b9532a0 R08: 000055573b992950 R09: 000055573b9a1ad0 May 3 16:48:05 nas kernel: [31727.491161] R10: 000000003b7c74ed R11: 0000000000000206 R12: 000055573b995c10 May 3 16:48:05 nas kernel: [31727.491164] R13: 000055573b97b2f0 R14: 000055573b968020 R15: 000055573b959cc0 May 3 16:48:05 nas kernel: [31727.491167] </TASK> May 3 16:48:05 nas kernel: [31727.491169] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1972898/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
