My experiments was here: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1773392
I installed new 18.04 + zfs + lxd - zfs hangs (well, that would be awkward if would not) +zfsutils-linux=0.7.5-1ubuntu16.3 - zfs hangs (ok, we need upgrade kernel too) +kernel=4.15.0.28 (all packages) - still hangs :( Am I missing something? +everything_0.7.5=0.7.5-1ubuntu16.3 - still hangs... +upgrade+dist-upgrade from proposed - still hangs... So this bug doesn't fixed for me... And it is strange, because I used to install zfs 0.7.9 from source with commit that fixed bug (https://github.com/zfsonlinux/zfs/pull/7693) - and that worked for me on default kernel (4.15.0.20). -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to zfs-linux in Ubuntu. https://bugs.launchpad.net/bugs/1781364 Title: Kernel error "task zfs:pid blocked for more than 120 seconds" Status in Linux: Fix Released Status in linux package in Ubuntu: Fix Committed Status in zfs-linux package in Ubuntu: Fix Released Status in linux source package in Xenial: Confirmed Status in zfs-linux source package in Xenial: Confirmed Status in linux source package in Bionic: Confirmed Status in zfs-linux source package in Bionic: Fix Committed Status in linux source package in Cosmic: Fix Committed Status in zfs-linux source package in Cosmic: Fix Released Bug description: == SRU Justification, XENIAL, BIONIC == Exercising ZFS with lxd with many mount/umounts can cause lockups and 120 second timeout messages. == How to reproduce bug == In a VM, 2 CPUs, 16GB of memory running Bionic: sudo apt update sudo apt install lxd lxd-client lxd-tools zfsutils-linux sudo lxd init (and with the default init options) then run: lxd-benchmark launch --count 96 --parallel 96 This will reliably show the lockup every time without the fix. With the fix (detailed below) one cannot reproduce the lockup. == Fix == Upstream ZFS commit commit ac09630d8b0bf6c92084a30fdaefd03fd0adbdc1 Author: Brian Behlendorf <behlendo...@llnl.gov> Date: Wed Jul 11 15:49:10 2018 -0700 Fix zpl_mount() deadlock == Regression Potential == This just changes the locking in the mount path of ZFS and will only affect ZFS mount/unmounts. The regression potential is small as this touches a very small code path that has been exhaustively exercises this code under multiple thread/CPU contention and shown not to break. ------------------ ZFS bug report: https://github.com/zfsonlinux/zfs/issues/7691 "I am using LXD containers that are configured to use a ZFS storage backend. I create many containers using a benchmark tool, which probably stresses the use of ZFS. In two out of four attempts, I got [ 725.970508] INFO: task lxd:4455 blocked for more than 120 seconds. [ 725.976730] Tainted: P O 4.15.0-20-generic #21-Ubuntu [ 725.983551] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 725.991624] INFO: task txg_sync:4202 blocked for more than 120 seconds. [ 725.998264] Tainted: P O 4.15.0-20-generic #21-Ubuntu [ 726.005071] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 726.013313] INFO: task lxd:99919 blocked for more than 120 seconds. [ 726.019609] Tainted: P O 4.15.0-20-generic #21-Ubuntu [ 726.026418] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 726.034560] INFO: task zfs:100513 blocked for more than 120 seconds. [ 726.040936] Tainted: P O 4.15.0-20-generic #21-Ubuntu [ 726.047746] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 726.055791] INFO: task zfs:100584 blocked for more than 120 seconds. [ 726.062170] Tainted: P O 4.15.0-20-generic #21-Ubuntu [ 726.068979] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Describe how to reproduce the problem Start an Ubuntu 18.04 LTS server. Install LXD if not already installed. sudo apt update sudo apt install lxd lxd-client lxd-tools zfsutils-linux Configure LXD with sudo lxd init. When prompted for the storage backend, select ZFS and specify an empty disk. $ sudo lxd init Would you like to use LXD clustering? (yes/no) [default=no]: Do you want to configure a new storage pool? (yes/no) [default=yes]: Name of the new storage pool [default=default]: Name of the storage backend to use (dir, zfs) [default=zfs]: Create a new ZFS pool? (yes/no) [default=yes]: Would you like to use an existing block device? (yes/no) [default=no]: yes Path to the existing block device: /dev/sdb Would you like to connect to a MAAS server? (yes/no) [default=no]: Would you like to create a new local network bridge? (yes/no) [default=yes]: no Would you like to configure LXD to use an existing bridge or host interface? (yes/no) [default=no]: no Would you like LXD to be available over the network? (yes/no) [default=no]: Would you like stale cached images to be updated automatically? (yes/no) [default=yes] Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: Now run the following to launch 48 containers in batches of 12. lxd-benchmark launch --count 48 --parallel 12 In two out of four attempts, I got the kernel errors. I also tried echo 1 >/sys/module/spl/parameters/spl_taskq_kick but did not manage to continue. Include any warning/errors/backtraces from the system logs dmesg output [ 725.970508] INFO: task lxd:4455 blocked for more than 120 seconds. [ 725.976730] Tainted: P O 4.15.0-20-generic #21-Ubuntu [ 725.983551] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 725.991408] lxd D 0 4455 1 0x00000000 [ 725.991412] Call Trace: [ 725.991424] __schedule+0x297/0x8b0 [ 725.991428] schedule+0x2c/0x80 [ 725.991429] rwsem_down_write_failed+0x162/0x360 [ 725.991460] ? dbuf_rele_and_unlock+0x1a8/0x4b0 [zfs] [ 725.991465] call_rwsem_down_write_failed+0x17/0x30 [ 725.991468] ? call_rwsem_down_write_failed+0x17/0x30 [ 725.991469] down_write+0x2d/0x40 [ 725.991472] grab_super+0x30/0x90 [ 725.991501] ? zpl_create+0x160/0x160 [zfs] [ 725.991504] sget_userns+0x91/0x490 [ 725.991507] ? get_anon_bdev+0x100/0x100 [ 725.991534] ? zpl_create+0x160/0x160 [zfs] [ 725.991537] sget+0x7d/0xa0 [ 725.991540] ? get_anon_bdev+0x100/0x100 [ 725.991567] zpl_mount+0xa8/0x160 [zfs] [ 725.991570] mount_fs+0x37/0x150 [ 725.991574] vfs_kern_mount.part.23+0x5d/0x110 [ 725.991576] do_mount+0x5ed/0xce0 [ 725.991577] ? copy_mount_options+0x2c/0x220 [ 725.991578] SyS_mount+0x98/0xe0 [ 725.991582] do_syscall_64+0x73/0x130 [ 725.991583] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 725.991585] RIP: 0033:0x4dbd5a [ 725.991586] RSP: 002b:000000c428be6890 EFLAGS: 00000206 ORIG_RAX: 00000000000000a5 [ 725.991588] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00000000004dbd5a [ 725.991589] RDX: 000000c421a04b7c RSI: 000000c426f94f40 RDI: 000000c4274ceaa0 [ 725.991590] RBP: 000000c428be6930 R08: 000000c425521a90 R09: 0000000000000000 [ 725.991590] R10: 0000000000000000 R11: 0000000000000206 R12: ffffffffffffffff [ 725.991591] R13: 000000000000003e R14: 000000000000003d R15: 0000000000000080 [ 725.991624] INFO: task txg_sync:4202 blocked for more than 120 seconds. [ 725.998264] Tainted: P O 4.15.0-20-generic #21-Ubuntu [ 726.005071] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 726.012928] txg_sync D 0 4202 2 0x80000000 [ 726.012930] Call Trace: [ 726.012933] __schedule+0x297/0x8b0 [ 726.012939] schedule+0x2c/0x80 [ 726.012945] cv_wait_common+0x11e/0x140 [spl] [ 726.012948] ? wait_woken+0x80/0x80 [ 726.012954] __cv_wait+0x15/0x20 [spl] [ 726.012981] rrw_enter_write+0x3c/0xa0 [zfs] [ 726.013006] rrw_enter+0x13/0x20 [zfs] [ 726.013033] spa_sync+0x7c9/0xd80 [zfs] [ 726.013062] txg_sync_thread+0x2cd/0x4a0 [zfs] [ 726.013089] ? txg_quiesce_thread+0x3d0/0x3d0 [zfs] [ 726.013093] thread_generic_wrapper+0x74/0x90 [spl] [ 726.013098] kthread+0x121/0x140 [ 726.013101] ? __thread_exit+0x20/0x20 [spl] [ 726.013103] ? kthread_create_worker_on_cpu+0x70/0x70 [ 726.013104] ret_from_fork+0x22/0x40 [ 726.013313] INFO: task lxd:99919 blocked for more than 120 seconds. [ 726.019609] Tainted: P O 4.15.0-20-generic #21-Ubuntu [ 726.026418] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 726.034272] lxd D 0 99919 99626 0x00000000 [ 726.034274] Call Trace: [ 726.034277] __schedule+0x297/0x8b0 [ 726.034283] ? __wake_up_common+0x73/0x130 [ 726.034286] schedule+0x2c/0x80 [ 726.034290] cv_wait_common+0x11e/0x140 [spl] [ 726.034293] ? wait_woken+0x80/0x80 [ 726.034297] __cv_wait+0x15/0x20 [spl] [ 726.034322] txg_wait_synced+0xdd/0x130 [zfs] [ 726.034349] zil_create+0x249/0x300 [zfs] [ 726.034376] zil_commit_writer+0x6d1/0x7c0 [zfs] [ 726.034401] zil_commit.part.14+0x93/0x100 [zfs] [ 726.034429] zil_commit+0x17/0x20 [zfs] [ 726.034457] zfs_sync+0x6e/0xb0 [zfs] [ 726.034484] zpl_sync_fs+0x50/0x80 [zfs] [ 726.034490] __sync_filesystem+0x33/0x60 [ 726.034493] sync_filesystem+0x39/0x40 [ 726.034495] generic_shutdown_super+0x27/0x120 [ 726.034496] kill_anon_super+0x12/0x20 [ 726.034518] zpl_kill_sb+0x1a/0x20 [zfs] [ 726.034524] deactivate_locked_super+0x48/0x80 [ 726.034529] deactivate_super+0x40/0x60 [ 726.034532] cleanup_mnt+0x3f/0x80 [ 726.034534] __cleanup_mnt+0x12/0x20 [ 726.034535] task_work_run+0x9d/0xc0 [ 726.034537] exit_to_usermode_loop+0xc0/0xd0 [ 726.034539] do_syscall_64+0x115/0x130 [ 726.034542] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 726.034550] RIP: 0033:0x7fb553b3e8c7 [ 726.034551] RSP: 002b:00007fff426c5388 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 726.034553] RAX: 0000000000000000 RBX: 000000000000000f RCX: 00007fb553b3e8c7 [ 726.034553] RDX: 00007fb55476eb9f RSI: 0000000000000002 RDI: 00007fb554770b6b [ 726.034554] RBP: 000000000000000c R08: 0000000000000000 R09: 00007fb553b8ae67 [ 726.034555] R10: 0000000000084000 R11: 0000000000000246 R12: 00007fff426c64a0 [ 726.034555] R13: 0000000003176690 R14: 0000000003177810 R15: 00000000031741f0 [ 726.034560] INFO: task zfs:100513 blocked for more than 120 seconds. [ 726.040936] Tainted: P O 4.15.0-20-generic #21-Ubuntu [ 726.047746] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 726.055600] zfs D 0 100513 2626 0x00000000 [ 726.055602] Call Trace: [ 726.055606] __schedule+0x297/0x8b0 [ 726.055609] schedule+0x2c/0x80 [ 726.055613] cv_wait_common+0x11e/0x140 [spl] [ 726.055615] ? wait_woken+0x80/0x80 [ 726.055619] __cv_wait+0x15/0x20 [spl] [ 726.055642] rrw_enter_read_impl+0x4e/0x160 [zfs] [ 726.055666] rrw_enter+0x1c/0x20 [zfs] [ 726.055691] dsl_pool_hold+0x5a/0x80 [zfs] [ 726.055713] dmu_objset_hold+0x33/0xa0 [zfs] [ 726.055740] zfs_ioc_objset_stats+0x32/0xa0 [zfs] [ 726.055766] zfsdev_ioctl+0x1e0/0x610 [zfs] [ 726.055771] do_vfs_ioctl+0xa8/0x630 [ 726.055774] ? handle_mm_fault+0xb1/0x1f0 [ 726.055776] ? __do_page_fault+0x270/0x4d0 [ 726.055777] SyS_ioctl+0x79/0x90 [ 726.055779] do_syscall_64+0x73/0x130 [ 726.055781] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 726.055782] RIP: 0033:0x7fd4adc795d7 [ 726.055782] RSP: 002b:00007ffe356b8718 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [ 726.055783] RAX: ffffffffffffffda RBX: 00007ffe356b8740 RCX: 00007fd4adc795d7 [ 726.055784] RDX: 00007ffe356b8740 RSI: 0000000000005a12 RDI: 0000000000000003 [ 726.055785] RBP: 000055632e278660 R08: 000000000000ffff R09: 00007fd4adcd1ed0 [ 726.055785] R10: 2f746c7561666564 R11: 0000000000000246 R12: 000055632e278660 [ 726.055788] R13: 00007ffe356beec0 R14: 00007fd4af1756e0 R15: 00007ffe356bbe30 [ 726.055791] INFO: task zfs:100584 blocked for more than 120 seconds. [ 726.062170] Tainted: P O 4.15.0-20-generic #21-Ubuntu [ 726.068979] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 726.076835] zfs D 0 100584 15513 0x00000000 [ 726.076837] Call Trace: [ 726.076840] __schedule+0x297/0x8b0 [ 726.076845] schedule+0x2c/0x80 [ 726.076851] cv_wait_common+0x11e/0x140 [spl] [ 726.076854] ? wait_woken+0x80/0x80 [ 726.076859] __cv_wait+0x15/0x20 [spl] [ 726.076881] rrw_enter_read_impl+0x4e/0x160 [zfs] [ 726.076905] rrw_enter+0x1c/0x20 [zfs] [ 726.076929] dsl_pool_hold+0x5a/0x80 [zfs] [ 726.076951] dmu_objset_hold+0x33/0xa0 [zfs] [ 726.076977] zfs_ioc_objset_stats+0x32/0xa0 [zfs] [ 726.077001] zfsdev_ioctl+0x1e0/0x610 [zfs] [ 726.077005] do_vfs_ioctl+0xa8/0x630 [ 726.077006] ? handle_mm_fault+0xb1/0x1f0 [ 726.077008] ? __do_page_fault+0x270/0x4d0 [ 726.077010] SyS_ioctl+0x79/0x90 [ 726.077011] do_syscall_64+0x73/0x130 [ 726.077013] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 726.077014] RIP: 0033:0x7fc2734075d7 [ 726.077014] RSP: 002b:00007fff653a4b08 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [ 726.077015] RAX: ffffffffffffffda RBX: 00007fff653a4b30 RCX: 00007fc2734075d7 [ 726.077016] RDX: 00007fff653a4b30 RSI: 0000000000005a12 RDI: 0000000000000003 [ 726.077017] RBP: 000055f3576e9660 R08: 000000000000ffff R09: 00007fc27345fed0 [ 726.077017] R10: 2f746c7561666564 R11: 0000000000000246 R12: 000055f3576e9660 [ 726.077018] R13: 00007fff653aaec0 R14: 00007fc2749036e0 R15: 00007fff653a8220 [ 846.801124] INFO: task lxd:4455 blocked for more than 120 seconds. [ 846.807352] Tainted: P O 4.15.0-20-generic #21-Ubuntu [ 846.814170] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 846.822028] lxd D 0 4455 1 0x00000000 [ 846.822031] Call Trace: [ 846.822042] __schedule+0x297/0x8b0 [ 846.822045] schedule+0x2c/0x80 [ 846.822047] rwsem_down_write_failed+0x162/0x360 [ 846.822078] ? dbuf_rele_and_unlock+0x1a8/0x4b0 [zfs] [ 846.822083] call_rwsem_down_write_failed+0x17/0x30 [ 846.822086] ? call_rwsem_down_write_failed+0x17/0x30 [ 846.822087] down_write+0x2d/0x40 [ 846.822091] grab_super+0x30/0x90 [ 846.822118] ? zpl_create+0x160/0x160 [zfs] [ 846.822121] sget_userns+0x91/0x490 [ 846.822123] ? get_anon_bdev+0x100/0x100 [ 846.822150] ? zpl_create+0x160/0x160 [zfs] [ 846.822153] sget+0x7d/0xa0 [ 846.822156] ? get_anon_bdev+0x100/0x100 [ 846.822181] zpl_mount+0xa8/0x160 [zfs] [ 846.822183] mount_fs+0x37/0x150 [ 846.822188] vfs_kern_mount.part.23+0x5d/0x110 [ 846.822189] do_mount+0x5ed/0xce0 [ 846.822190] ? copy_mount_options+0x2c/0x220 [ 846.822192] SyS_mount+0x98/0xe0 [ 846.822195] do_syscall_64+0x73/0x130 [ 846.822196] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 846.822198] RIP: 0033:0x4dbd5a [ 846.822199] RSP: 002b:000000c428be6890 EFLAGS: 00000206 ORIG_RAX: 00000000000000a5 [ 846.822201] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00000000004dbd5a [ 846.822203] RDX: 000000c421a04b7c RSI: 000000c426f94f40 RDI: 000000c4274ceaa0 [ 846.822205] RBP: 000000c428be6930 R08: 000000c425521a90 R09: 0000000000000000 [ 846.822206] R10: 0000000000000000 R11: 0000000000000206 R12: ffffffffffffffff [ 846.822206] R13: 000000000000003e R14: 000000000000003d R15: 0000000000000080 [ 846.822239] INFO: task txg_sync:4202 blocked for more than 120 seconds. [ 846.828882] Tainted: P O 4.15.0-20-generic #21-Ubuntu [ 846.835692] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 846.843549] txg_sync D 0 4202 2 0x80000000 [ 846.843551] Call Trace: [ 846.843554] __schedule+0x297/0x8b0 [ 846.843560] schedule+0x2c/0x80 [ 846.843566] cv_wait_common+0x11e/0x140 [spl] [ 846.843570] ? wait_woken+0x80/0x80 [ 846.843574] __cv_wait+0x15/0x20 [spl] [ 846.843603] rrw_enter_write+0x3c/0xa0 [zfs] [ 846.843629] rrw_enter+0x13/0x20 [zfs] [ 846.843654] spa_sync+0x7c9/0xd80 [zfs] [ 846.843682] txg_sync_thread+0x2cd/0x4a0 [zfs] [ 846.843708] ? txg_quiesce_thread+0x3d0/0x3d0 [zfs] [ 846.843713] thread_generic_wrapper+0x74/0x90 [spl] [ 846.843717] kthread+0x121/0x140 [ 846.843720] ? __thread_exit+0x20/0x20 [spl] [ 846.843721] ? kthread_create_worker_on_cpu+0x70/0x70 [ 846.843723] ret_from_fork+0x22/0x40 [ 846.843931] INFO: task lxd:99919 blocked for more than 120 seconds. [ 846.850227] Tainted: P O 4.15.0-20-generic #21-Ubuntu [ 846.857040] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 846.864892] lxd D 0 99919 99626 0x00000000 [ 846.864894] Call Trace: [ 846.864897] __schedule+0x297/0x8b0 [ 846.864903] ? __wake_up_common+0x73/0x130 [ 846.864906] schedule+0x2c/0x80 [ 846.864910] cv_wait_common+0x11e/0x140 [spl] [ 846.864912] ? wait_woken+0x80/0x80 [ 846.864917] __cv_wait+0x15/0x20 [spl] [ 846.864942] txg_wait_synced+0xdd/0x130 [zfs] [ 846.864971] zil_create+0x249/0x300 [zfs] [ 846.864998] zil_commit_writer+0x6d1/0x7c0 [zfs] [ 846.865023] zil_commit.part.14+0x93/0x100 [zfs] [ 846.865051] zil_commit+0x17/0x20 [zfs] [ 846.865080] zfs_sync+0x6e/0xb0 [zfs] [ 846.865107] zpl_sync_fs+0x50/0x80 [zfs] [ 846.865111] __sync_filesystem+0x33/0x60 [ 846.865113] sync_filesystem+0x39/0x40 [ 846.865114] generic_shutdown_super+0x27/0x120 [ 846.865116] kill_anon_super+0x12/0x20 [ 846.865138] zpl_kill_sb+0x1a/0x20 [zfs] [ 846.865140] deactivate_locked_super+0x48/0x80 [ 846.865143] deactivate_super+0x40/0x60 [ 846.865145] cleanup_mnt+0x3f/0x80 [ 846.865147] __cleanup_mnt+0x12/0x20 [ 846.865148] task_work_run+0x9d/0xc0 [ 846.865150] exit_to_usermode_loop+0xc0/0xd0 [ 846.865152] do_syscall_64+0x115/0x130 [ 846.865153] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 846.865154] RIP: 0033:0x7fb553b3e8c7 [ 846.865155] RSP: 002b:00007fff426c5388 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 846.865156] RAX: 0000000000000000 RBX: 000000000000000f RCX: 00007fb553b3e8c7 [ 846.865158] RDX: 00007fb55476eb9f RSI: 0000000000000002 RDI: 00007fb554770b6b [ 846.865159] RBP: 000000000000000c R08: 0000000000000000 R09: 00007fb553b8ae67 [ 846.865162] R10: 0000000000084000 R11: 0000000000000246 R12: 00007fff426c64a0 [ 846.865162] R13: 0000000003176690 R14: 0000000003177810 R15: 00000000031741f0 [ 846.865167] INFO: task zfs:100513 blocked for more than 120 seconds. [ 846.871546] Tainted: P O 4.15.0-20-generic #21-Ubuntu [ 846.878357] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 846.886214] zfs D 0 100513 2626 0x00000000 [ 846.886215] Call Trace: [ 846.886218] __schedule+0x297/0x8b0 [ 846.886223] schedule+0x2c/0x80 [ 846.886230] cv_wait_common+0x11e/0x140 [spl] [ 846.886232] ? wait_woken+0x80/0x80 [ 846.886237] __cv_wait+0x15/0x20 [spl] [ 846.886261] rrw_enter_read_impl+0x4e/0x160 [zfs] [ 846.886284] rrw_enter+0x1c/0x20 [zfs] [ 846.886309] dsl_pool_hold+0x5a/0x80 [zfs] [ 846.886331] dmu_objset_hold+0x33/0xa0 [zfs] [ 846.886357] zfs_ioc_objset_stats+0x32/0xa0 [zfs] [ 846.886383] zfsdev_ioctl+0x1e0/0x610 [zfs] [ 846.886388] do_vfs_ioctl+0xa8/0x630 [ 846.886391] ? handle_mm_fault+0xb1/0x1f0 [ 846.886394] ? __do_page_fault+0x270/0x4d0 [ 846.886396] SyS_ioctl+0x79/0x90 [ 846.886397] do_syscall_64+0x73/0x130 [ 846.886399] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 846.886400] RIP: 0033:0x7fd4adc795d7 [ 846.886402] RSP: 002b:00007ffe356b8718 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [ 846.886405] RAX: ffffffffffffffda RBX: 00007ffe356b8740 RCX: 00007fd4adc795d7 [ 846.886406] RDX: 00007ffe356b8740 RSI: 0000000000005a12 RDI: 0000000000000003 [ 846.886406] RBP: 000055632e278660 R08: 000000000000ffff R09: 00007fd4adcd1ed0 [ 846.886407] R10: 2f746c7561666564 R11: 0000000000000246 R12: 000055632e278660 [ 846.886408] R13: 00007ffe356beec0 R14: 00007fd4af1756e0 R15: 00007ffe356bbe30 [ 846.886410] INFO: task zfs:100584 blocked for more than 120 seconds. [ 846.892790] Tainted: P O 4.15.0-20-generic #21-Ubuntu [ 846.899598] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 846.907454] zfs D 0 100584 15513 0x00000000 [ 846.907456] Call Trace: [ 846.907459] __schedule+0x297/0x8b0 [ 846.907465] schedule+0x2c/0x80 [ 846.907470] cv_wait_common+0x11e/0x140 [spl] [ 846.907471] ? wait_woken+0x80/0x80 [ 846.907477] __cv_wait+0x15/0x20 [spl] [ 846.907499] rrw_enter_read_impl+0x4e/0x160 [zfs] [ 846.907523] rrw_enter+0x1c/0x20 [zfs] [ 846.907547] dsl_pool_hold+0x5a/0x80 [zfs] [ 846.907568] dmu_objset_hold+0x33/0xa0 [zfs] [ 846.907594] zfs_ioc_objset_stats+0x32/0xa0 [zfs] [ 846.907618] zfsdev_ioctl+0x1e0/0x610 [zfs] [ 846.907622] do_vfs_ioctl+0xa8/0x630 [ 846.907624] ? handle_mm_fault+0xb1/0x1f0 [ 846.907625] ? __do_page_fault+0x270/0x4d0 [ 846.907627] SyS_ioctl+0x79/0x90 [ 846.907628] do_syscall_64+0x73/0x130 [ 846.907630] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 846.907631] RIP: 0033:0x7fc2734075d7 [ 846.907631] RSP: 002b:00007fff653a4b08 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [ 846.907633] RAX: ffffffffffffffda RBX: 00007fff653a4b30 RCX: 00007fc2734075d7 [ 846.907633] RDX: 00007fff653a4b30 RSI: 0000000000005a12 RDI: 0000000000000003 [ 846.907634] RBP: 000055f3576e9660 R08: 000000000000ffff R09: 00007fc27345fed0 [ 846.907635] R10: 2f746c7561666564 R11: 0000000000000246 R12: 000055f3576e9660 [ 846.907637] R13: 00007fff653aaec0 R14: 00007fc2749036e0 R15: 00007fff653a8220 Contents of "/proc/spl/kstat/zfs/arcstats" 13 1 0x01 96 4608 11300672527 1808059980062 name type data hits 4 44186496 misses 4 1247761 demand_data_hits 4 3327097 demand_data_misses 4 17953 demand_metadata_hits 4 39832270 demand_metadata_misses 4 1076950 prefetch_data_hits 4 1357 prefetch_data_misses 4 23 prefetch_metadata_hits 4 1025772 prefetch_metadata_misses 4 152835 mru_hits 4 18193851 mru_ghost_hits 4 0 mfu_hits 4 24976976 mfu_ghost_hits 4 0 deleted 4 10 mutex_miss 4 0 access_skip 4 68 evict_skip 4 1 evict_not_enough 4 0 evict_l2_cached 4 0 evict_l2_eligible 4 101376 evict_l2_ineligible 4 2048 evict_l2_skip 4 0 hash_elements 4 38683 hash_elements_max 4 57741 hash_collisions 4 1520 hash_chains 4 5 hash_chain_max 4 1 p 4 16815604736 c 4 33631209472 c_min 4 2101950592 c_max 4 33631209472 size 4 833199872 compressed_size 4 286609408 uncompressed_size 4 831992320 overhead_size 4 330568192 hdr_size 4 14660144 data_size 4 193579520 metadata_size 4 423598080 dbuf_size 4 46065424 dnode_size 4 112384064 bonus_size 4 42912640 anon_size 4 36502016 anon_evictable_data 4 0 anon_evictable_metadata 4 0 mru_size 4 310932480 mru_evictable_data 4 27623424 mru_evictable_metadata 4 136801792 mru_ghost_size 4 0 mru_ghost_evictable_data 4 0 mru_ghost_evictable_metadata 4 0 mfu_size 4 269743104 mfu_evictable_data 4 53612032 mfu_evictable_metadata 4 3195392 mfu_ghost_size 4 0 mfu_ghost_evictable_data 4 0 mfu_ghost_evictable_metadata 4 0 l2_hits 4 0 l2_misses 4 0 l2_feeds 4 0 l2_rw_clash 4 0 l2_read_bytes 4 0 l2_write_bytes 4 0 l2_writes_sent 4 0 l2_writes_done 4 0 l2_writes_error 4 0 l2_writes_lock_retry 4 0 l2_evict_lock_retry 4 0 l2_evict_reading 4 0 l2_evict_l1cached 4 0 l2_free_on_write 4 0 l2_abort_lowmem 4 0 l2_cksum_bad 4 0 l2_io_error 4 0 l2_size 4 0 l2_asize 4 0 l2_hdr_size 4 0 memory_throttle_count 4 0 memory_direct_count 4 0 memory_indirect_count 4 0 memory_all_bytes 4 67262418944 memory_free_bytes 4 57157578752 memory_available_bytes 3 56106606592 arc_no_grow 4 0 arc_tempreserve 4 0 arc_loaned_bytes 4 0 arc_prune 4 0 arc_meta_used 4 639620352 arc_meta_limit 4 25223407104 arc_dnode_limit 4 2522340710 arc_meta_max 4 965175896 arc_meta_min 4 16777216 sync_wait_for_async 4 168 demand_hit_predictive_prefetch 4 999 arc_need_free 4 0 arc_sys_free 4 1050975296 Command "slabtop -o" Active / Total Objects (% used) : 28354235 / 29140626 (97.3%) Active / Total Slabs (% used) : 382017 / 382017 (100.0%) Active / Total Caches (% used) : 95 / 128 (74.2%) Active / Total Size (% used) : 4580408.56K / 4743868.00K (96.6%) Minimum / Average / Maximum Object : 0.01K / 0.16K / 21.81K OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 17206144 17204917 0% 0.03K 134423 128 537692K kmalloc-32 1141392 966790 0% 0.19K 27176 42 217408K dentry 1114496 1004537 0% 0.06K 17414 64 69656K kmalloc-64 1088192 1078186 0% 0.50K 17003 64 544096K kmalloc-512 1038300 714247 0% 0.13K 17305 60 138440K kernfs_node_cache 937536 931296 0% 0.25K 14649 64 234384K filp 684160 682244 0% 0.06K 10690 64 42760K pid 582099 569877 0% 0.59K 10983 53 351456K inode_cache 520104 518529 0% 0.20K 13336 39 106688K vm_area_struct 404334 388371 0% 0.09K 9627 42 38508K kmalloc-96 342286 341596 0% 0.09K 7441 46 29764K anon_vma 342016 338948 0% 0.25K 5344 64 85504K kmalloc-256 277032 276822 0% 0.19K 6596 42 52768K cred_jar 248352 241634 0% 0.66K 5174 48 165568K proc_inode_cache 248320 233052 0% 0.01K 485 512 1940K kmalloc-8 214984 143177 0% 0.57K 3839 56 122848K radix_tree_node To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/1781364/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
