[Bug 1654517] Re: ZFS I/O hangs for minutes
It happened to me, is not usual, in fact was muy first time, in my case it hanged forever, i was making a heavy use with nfs, ocal and lot of paralel copy writes and it stucked. No problems with the disks, just the process using 99% of cpu. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654517 Title: ZFS I/O hangs for minutes To manage notifications about this bug go to: https://bugs.launchpad.net/zfs/+bug/1654517/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654517] Re: ZFS I/O hangs for minutes
Ok, thanks for changing the status, Colin. :) G -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654517 Title: ZFS I/O hangs for minutes To manage notifications about this bug go to: https://bugs.launchpad.net/zfs/+bug/1654517/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654517] Re: ZFS I/O hangs for minutes
This bug has been waiting for feedback for questions in comment #35 and #36 for months. I'm going to mark this bug as Won't fix because of limited responses. If this bug is still an issue, please re-open it. ** Changed in: zfs-linux (Ubuntu) Status: Incomplete => Won't Fix -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654517 Title: ZFS I/O hangs for minutes To manage notifications about this bug go to: https://bugs.launchpad.net/zfs/+bug/1654517/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654517] Re: ZFS I/O hangs for minutes
Hello Karl-Philipp, Thank you for submitting this bug and reporting a problem with the zfs-linux package. You made this bug report in 2007 and there have been several versions of Ubuntu since then. Could you confirm that this is no longer a problem and that we can close the ticket? If it is still a problem, are you still interested in finding a solution to this bug? If you are, could you let us know and, in the current version, could you run the following (only once): apport-collect 1654517 and upload the updated logs and and any other logs that are relevant for this particular issue. Thank you again for helping make Ubuntu better. G -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654517 Title: ZFS I/O hangs for minutes To manage notifications about this bug go to: https://bugs.launchpad.net/zfs/+bug/1654517/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654517] Re: ZFS I/O hangs for minutes
This stack trace is similar to one in https://github.com/zfsonlinux/zfs/issues/2934 - the suspected issue is that one of the devices is taking a long time to complete and ZFS is blocked waiting for this. I suggest installing sysstat sudo apt install sysstat and when the lockup happens run: iostat -mx so that we can see if it is a long device delay or not. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654517 Title: ZFS I/O hangs for minutes To manage notifications about this bug go to: https://bugs.launchpad.net/zfs/+bug/1654517/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654517] Re: ZFS I/O hangs for minutes
Found one in my Debian box logs: Jun 11 10:02:37 workstation kernel: [238276.640617] INFO: task txg_sync:5936 blocked for more than 120 seconds. Jun 11 10:02:37 workstation kernel: [238276.643806] Tainted: P O 4.16.0-2-amd64 #1 Debian 4.16.12- Jun 11 10:02:37 workstation kernel: [238276.646803] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this m Jun 11 10:02:37 workstation kernel: [238276.649819] txg_syncD0 5936 2 0x8000 Jun 11 10:02:37 workstation kernel: [238276.652850] Call Trace: Jun 11 10:02:37 workstation kernel: [238276.655776] ? __schedule+0x291/0x870 Jun 11 10:02:37 workstation kernel: [238276.658671] ? zio_taskq_dispatch+0x6f/0x90 [zfs] Jun 11 10:02:37 workstation kernel: [238276.661196] ? zio_nowait+0xa3/0x140 [zfs] Jun 11 10:02:37 workstation kernel: [238276.664199] schedule+0x28/0x80 Jun 11 10:02:37 workstation kernel: [238276.667206] io_schedule+0x12/0x40 Jun 11 10:02:37 workstation kernel: [238276.670226] cv_wait_common+0xac/0x130 [spl] Jun 11 10:02:37 workstation kernel: [238276.673206] ? finish_wait+0x80/0x80 Jun 11 10:02:37 workstation kernel: [238276.676522] zio_wait+0xe6/0x1a0 [zfs] Jun 11 10:02:37 workstation kernel: [238276.679588] dsl_pool_sync+0xe6/0x440 [zfs] Jun 11 10:02:37 workstation kernel: [238276.682620] spa_sync+0x424/0xcf0 [zfs] Jun 11 10:02:37 workstation kernel: [238276.685657] txg_sync_thread+0x2ce/0x490 [zfs] Jun 11 10:02:37 workstation kernel: [238276.688661] ? txg_delay+0x1b0/0x1b0 [zfs] Jun 11 10:02:37 workstation kernel: [238276.691641] ? __thread_exit+0x20/0x20 [spl] Jun 11 10:02:37 workstation kernel: [238276.694604] thread_generic_wrapper+0x6f/0x80 [spl] Jun 11 10:02:37 workstation kernel: [238276.697591] kthread+0x113/0x130 Jun 11 10:02:37 workstation kernel: [238276.700633] ? kthread_create_worker_on_cpu+0x70/0x70 Jun 11 10:02:37 workstation kernel: [238276.703594] ret_from_fork+0x22/0x40 Running zfs/spl 0.7.9-3. I think this still happens from time to time with mainline and lates zfs/spl (as I tested in another box). So, if its related to zfs cache dirty ratio, then the zfs_dirty_data_sync could make cache smaller and the sync wouldn't take so long.. if changing it doesn't solve, then its likely an intermittent locking issue on the zio_wait logic. Hope it helps! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654517 Title: ZFS I/O hangs for minutes To manage notifications about this bug go to: https://bugs.launchpad.net/zfs/+bug/1654517/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654517] Re: ZFS I/O hangs for minutes
Hello Colin, nope. It looks like ZFS has some sort of contention in waiting the ZIL to finish the async callbacks from scheduled work, even when more spindles exist. Not sure if its related to slab merging or getting the objects from either slab/kmalloc or directly from page allocs. I think its a locking contention due to zfs scalability iself, to be honest :\. I'm getting, sometimes, soft lockups here and there (all waiting on on zfs sync - txg_wait_synced). I'm using a raid-z with 4x480G SSDs and an extra SSD as L2ARC disk. Its fairly easy to reproduce, i'm basically compiling kernel tress all day in this machine, using the zfs pools as ccache's cache and as the compiled objects repository. I haven't explored tuning it though, considering its cache is not kept together with the pagecache (like changing zfs_dirty_data_sync or zfs_sync_taskq_batch_pct to make zfs cache flush less async, which might help). Cheers! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654517 Title: ZFS I/O hangs for minutes To manage notifications about this bug go to: https://bugs.launchpad.net/zfs/+bug/1654517/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654517] Re: ZFS I/O hangs for minutes
@Rafael, did the workaround in comment #31 help? ** Changed in: zfs-linux (Ubuntu) Status: Confirmed => Incomplete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654517 Title: ZFS I/O hangs for minutes To manage notifications about this bug go to: https://bugs.launchpad.net/zfs/+bug/1654517/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654517] Re: ZFS I/O hangs for minutes
You may possibly get some benefit from setting spl_kmem_cache_slab_limit to zero as a module parameter for the spl ZoL kernel module. This avoids SLUB slab merging entirely and maybe helpful from anecdotal evidence I've seen. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654517 Title: ZFS I/O hangs for minutes To manage notifications about this bug go to: https://bugs.launchpad.net/zfs/+bug/1654517/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654517] Re: ZFS I/O hangs for minutes
@Colin 1. its a really simple one since this is my devel machine: one ssd disk only zpool: https://pastebin.ubuntu.com/p/gZPtrxRDnF/ hdparm: https://pastebin.ubuntu.com/p/JYyGwQ7cVr/ 2. 8 x 4.0GHz AMD cores and 32GB RAM meminfo: https://pastebin.ubuntu.com/p/kSjXSC6R3q/ (at the time of soft lockup) 3. likely it does since its a single spindle for zpool iostat: https://pastebin.ubuntu.com/p/vbxXZKzmBp/ >From the stack trace, zfs seems to be setting a property in a newly created zfs filesystem (since i'm mostly creating containers at this specific point in time). Following the code, zfs_ioctl_init sets as a callback to ZFS_IOC_SET_PROP, the function zfs_ioc_set_prop() which will trigger the code what gets soft locked by 120 sec (spinning on txg_wait_synced). The txg kthread, likely the one responsible for sync to be finished, is waiting on completion for zio that were submitted. So, yes, I do agree that this might be an overload because of the amount of spindles/throughput I have. I have to check now if 4.4 is also in this same step because it looked dead locked to me, for hours. Upstream kernel/zfs/spl is, at least, in good conditions after I/O is finished and sync is done. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654517 Title: ZFS I/O hangs for minutes To manage notifications about this bug go to: https://bugs.launchpad.net/zfs/+bug/1654517/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654517] Re: ZFS I/O hangs for minutes
@Rafael, 1. what is your ZFS configuration (number of drives?) 2. size of your system (total memory, free memory). 3. Do your drive(s) suffer from large I/O latencies? - what is the output from iostat -mx -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654517 Title: ZFS I/O hangs for minutes To manage notifications about this bug go to: https://bugs.launchpad.net/zfs/+bug/1654517/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654517] Re: ZFS I/O hangs for minutes
Same issue (apparently) on upstream bug report tool: https://github.com/zfsonlinux/zfs/issues/2611 https://github.com/zfsonlinux/zfs/issues/2523 https://github.com/zfsonlinux/zfs/issues/4011 https://github.com/zfsonlinux/zfs/issues/2934 ** Bug watch added: Github Issue Tracker for ZFS #2611 https://github.com/zfsonlinux/zfs/issues/2611 ** Bug watch added: Github Issue Tracker for ZFS #2523 https://github.com/zfsonlinux/zfs/issues/2523 ** Bug watch added: Github Issue Tracker for ZFS #4011 https://github.com/zfsonlinux/zfs/issues/4011 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654517 Title: ZFS I/O hangs for minutes To manage notifications about this bug go to: https://bugs.launchpad.net/zfs/+bug/1654517/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654517] Re: ZFS I/O hangs for minutes
Colin, I'm afraid this is an ongoing issue. I have faced this in 4.4 and 4.13 kernels: [ 240.568146] INFO: task txg_sync:2065 blocked for more than 120 seconds. [ 240.568218] Tainted: P O4.4.0-116-generic #140-Ubuntu [ 240.568268] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 240.568290] txg_syncD 880739e63aa8 0 2065 2 0x [ 240.568294] 880739e63aa8 880739e63a88 88081af75400 8800ac635400 [ 240.568296] 880739e64000 88083ed972c0 7fff 88009fffc968 [ 240.568297] 0001 880739e63ac0 8184ae45 [ 240.568300] Call Trace: [ 240.568305] [] schedule+0x35/0x80 [ 240.568307] [] schedule_timeout+0x1b8/0x270 [ 240.568310] [] ? default_wake_function+0x12/0x20 [ 240.568313] [] ? __wake_up_common+0x58/0x90 [ 240.568315] [] ? ktime_get+0x3e/0xb0 [ 240.568317] [] io_schedule_timeout+0xa4/0x110 [ 240.568325] [] cv_wait_common+0xbc/0x140 [spl] [ 240.568327] [] ? wake_atomic_t_function+0x60/0x60 [ 240.568332] [] __cv_wait_io+0x18/0x20 [spl] [ 240.568373] [] zio_wait+0x114/0x200 [zfs] [ 240.568397] [] dsl_pool_sync+0xb8/0x430 [zfs] [ 240.568424] [] spa_sync+0x366/0xb30 [zfs] [ 240.568426] [] ? default_wake_function+0x12/0x20 [ 240.568453] [] txg_sync_thread+0x3ba/0x630 [zfs] [ 240.568481] [] ? txg_delay+0x180/0x180 [zfs] [ 240.568486] [] ? __thread_exit+0x20/0x20 [spl] [ 240.568490] [] thread_generic_wrapper+0x73/0x80 [spl] [ 240.568492] [] kthread+0xe7/0x100 [ 240.568494] [] ? kthread_create_on_node+0x1e0/0x1e0 [ 240.568495] [] ret_from_fork+0x42/0x80 [ 240.568497] [] ? kthread_create_on_node+0x1e0/0x1e0 [ 240.568505] INFO: task lxd:2562 blocked for more than 120 seconds. [ 240.568524] Tainted: P O4.4.0-116-generic #140-Ubuntu [ 240.568543] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 240.568565] lxd D 880734773ba0 0 2562 1 0x [ 240.568567] 880734773ba0 0246 88081af71c00 88080d18f000 [ 240.568569] 880734774000 8808144dba20 8808144dba48 8808144dbae0 [ 240.568571] 880734773bb8 8184ae45 8808144dbad8 [ 240.568573] Call Trace: [ 240.568575] [] schedule+0x35/0x80 [ 240.568579] [] cv_wait_common+0x10b/0x140 [spl] [ 240.568581] [] ? wake_atomic_t_function+0x60/0x60 [ 240.568586] [] __cv_wait+0x15/0x20 [spl] [ 240.568614] [] txg_wait_synced+0xe5/0x130 [zfs] [ 240.568643] [] zil_replay+0xdb/0x120 [zfs] [ 240.568673] [] zfs_sb_setup+0x16f/0x180 [zfs] [ 240.568702] [] zfs_domount+0x2c8/0x350 [zfs] [ 240.568731] [] ? zpl_kill_sb+0x20/0x20 [zfs] [ 240.568761] [] zpl_fill_super+0x2c/0x40 [zfs] [ 240.568763] [] mount_nodev+0x4f/0xa0 [ 240.568792] [] zpl_mount+0x4f/0x80 [zfs] [ 240.568794] [] mount_fs+0x3d/0x170 [ 240.568796] [] vfs_kern_mount+0x67/0x110 [ 240.568798] [] do_mount+0x25f/0xda0 [ 240.568801] [] SyS_mount+0x9f/0x100 [ 240.568803] [] entry_SYSCALL_64_fastpath+0x1c/0xbb [ 240.568810] INFO: task sync:7760 blocked for more than 120 seconds. [ 240.568828] Tainted: P O4.4.0-116-generic #140-Ubuntu [ 240.568848] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. And I am still facing this with latest kernel and latest SPL and ZFS code inaddy@workstation:~$ sudo cat /proc/2832/stack [<0>] __cv_timedwait_common+0x12c/0x170 [spl] [<0>] __cv_timedwait_io+0x19/0x20 [spl] [<0>] zio_wait+0x119/0x270 [zfs] [<0>] dsl_pool_sync+0xb8/0x420 [zfs] [<0>] spa_sync+0x43e/0xd30 [zfs] [<0>] txg_sync_thread+0x271/0x440 [zfs] [<0>] thread_generic_wrapper+0x74/0x90 [spl] [<0>] kthread+0x121/0x140 [<0>] ret_from_fork+0x22/0x40 [<0>] 0x Linux workstation 4.16.0-rc7 #1 SMP Wed Mar 28 09:21:33 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux ii kmod-spl-4.16.0-rc70.7.0-31 amd64spl kernel module(s) for 4.16.0-rc7 ii kmod-spl-devel 0.7.0-31 amd64spl kernel module(s) devel common ii kmod-spl-devel-4.16.0-rc7 0.7.0-31 amd64spl kernel module(s) devel for 4.16.0-rc7 ii kmod-zfs-4.16.0-rc70.7.0-390 amd64zfs kernel module(s) for 4.16.0-rc7 ii kmod-zfs-devel 0.7.0-390 amd64zfs kernel module(s) devel common ii kmod-zfs-devel-4.16.0-rc7 0.7.0-390 amd64zfs kernel module(s) devel for 4.16.0-rc7 ii libzfs20.7.0-390 amd64
[Bug 1654517] Re: ZFS I/O hangs for minutes
** Changed in: zfs-linux (Ubuntu) Status: Incomplete => Won't Fix -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654517 Title: ZFS I/O hangs for minutes To manage notifications about this bug go to: https://bugs.launchpad.net/zfs/+bug/1654517/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654517] Re: ZFS I/O hangs for minutes
Hi, if this is still and issue please let me know. I'm going to close the bug if I don't get any response in 1 week. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654517 Title: ZFS I/O hangs for minutes To manage notifications about this bug go to: https://bugs.launchpad.net/zfs/+bug/1654517/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654517] Re: ZFS I/O hangs for minutes
Is this still causing problems? Ive not heard any update on my question in comment #24, so I was wondering if this bug is still an issue. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654517 Title: ZFS I/O hangs for minutes To manage notifications about this bug go to: https://bugs.launchpad.net/zfs/+bug/1654517/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654517] Re: ZFS I/O hangs for minutes
** Changed in: zfs Status: Unknown => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654517 Title: ZFS I/O hangs for minutes To manage notifications about this bug go to: https://bugs.launchpad.net/zfs/+bug/1654517/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654517] Re: ZFS I/O hangs for minutes
When this issue occurs again, can you run the following command: iostat -mx ..and past the output into the bug. Thanks ** Bug watch added: Github Issue Tracker for ZFS #2934 https://github.com/zfsonlinux/zfs/issues/2934 ** Also affects: zfs via https://github.com/zfsonlinux/zfs/issues/2934 Importance: Unknown Status: Unknown ** Changed in: zfs-linux (Ubuntu) Importance: Undecided => Medium ** Changed in: zfs-linux (Ubuntu) Assignee: (unassigned) => Colin Ian King (colin-king) ** Changed in: zfs-linux (Ubuntu) Status: Confirmed => Incomplete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654517 Title: ZFS I/O hangs for minutes To manage notifications about this bug go to: https://bugs.launchpad.net/zfs/+bug/1654517/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654517] Re: ZFS I/O hangs for minutes
** Description changed: I/O for multiple programs, like `thunderbird`, `firefox`, etc., hangs - for minutes, `dmesg` contains + for minutes and approx. 100 `z_rd_int_[n]` and `z_wr_int_[n]` kernel + threads are created, `dmesg` contains - [ 9184.451606] INFO: task txg_sync:11471 blocked for more than 120 seconds. - [ 9184.451610] Tainted: P OE 4.8.0-32-generic #34-Ubuntu - [ 9184.451611] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. - [ 9184.451612] txg_syncD a240ab3a7aa8 0 11471 2 0x - [ 9184.451616] a240ab3a7aa8 00ffbb6ade1f a24095148000 a240e5ca5580 - [ 9184.451618] 0046 a240ab3a8000 a240ff359200 7fff - [ 9184.451620] a23d36cf9050 0001 a240ab3a7ac0 bbe96b15 - [ 9184.451621] Call Trace: - [ 9184.451627] [] schedule+0x35/0x80 - [ 9184.451628] [] schedule_timeout+0x22a/0x3f0 - [ 9184.451631] [] ? __switch_to+0x2ce/0x6c0 - [ 9184.451633] [] ? pick_next_task_fair+0x48c/0x4c0 - [ 9184.451635] [] ? ktime_get+0x41/0xb0 - [ 9184.451636] [] io_schedule_timeout+0xa4/0x110 - [ 9184.451644] [] cv_wait_common+0xb2/0x130 [spl] - [ 9184.451646] [] ? wake_atomic_t_function+0x60/0x60 - [ 9184.451650] [] __cv_wait_io+0x18/0x20 [spl] - [ 9184.451689] [] zio_wait+0xfd/0x1d0 [zfs] - [ 9184.451716] [] dsl_pool_sync+0xb8/0x480 [zfs] - [ 9184.451745] [] spa_sync+0x37f/0xb30 [zfs] - [ 9184.451747] [] ? default_wake_function+0x12/0x20 - [ 9184.451779] [] txg_sync_thread+0x3a5/0x600 [zfs] - [ 9184.451807] [] ? txg_delay+0x160/0x160 [zfs] - [ 9184.451811] [] thread_generic_wrapper+0x71/0x80 [spl] - [ 9184.451815] [] ? __thread_exit+0x20/0x20 [spl] - [ 9184.451817] [] kthread+0xd8/0xf0 - [ 9184.451819] [] ret_from_fork+0x1f/0x40 - [ 9184.451821] [] ? kthread_create_on_node+0x1e0/0x1e0 - [ 9184.451849] INFO: task mozStorage #2:21607 blocked for more than 120 seconds. - [ 9184.451851] Tainted: P OE 4.8.0-32-generic #34-Ubuntu - [ 9184.451852] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. - [ 9184.451853] mozStorage #2 D a23fe8a5bd38 0 21607 19750 0x0004 - [ 9184.451855] a23fe8a5bd38 00ffa240ee8feb40 a240ecf72ac0 a2403803b900 - [ 9184.451857] bc2c02f7 a23fe8a5c000 a240aa940828 a240aa940800 - [ 9184.451858] a240aa940980 a23fe8a5bd50 bbe96b15 - [ 9184.451860] Call Trace: - [ 9184.451861] [] schedule+0x35/0x80 - [ 9184.451866] [] cv_wait_common+0x110/0x130 [spl] - [ 9184.451868] [] ? wake_atomic_t_function+0x60/0x60 - [ 9184.451872] [] __cv_wait+0x15/0x20 [spl] - [ 9184.451904] [] zil_commit.part.11+0x79/0x7a0 [zfs] - [ 9184.451909] [] ? tsd_hash_search.isra.0+0x46/0xa0 [spl] - [ 9184.451913] [] ? tsd_set+0x2b4/0x500 [spl] - [ 9184.451914] [] ? mutex_lock+0x12/0x30 - [ 9184.451945] [] zil_commit+0x17/0x20 [zfs] - [ 9184.451975] [] zfs_fsync+0x7a/0xf0 [zfs] - [ 9184.452005] [] zpl_fsync+0x68/0xa0 [zfs] - [ 9184.452008] [] vfs_fsync_range+0x4b/0xb0 - [ 9184.452010] [] do_fsync+0x3d/0x70 - [ 9184.452011] [] SyS_fsync+0x10/0x20 - [ 9184.452013] [] entry_SYSCALL_64_fastpath+0x1e/0xa8 - [ 9184.452023] INFO: task bitcoin-msghand:663 blocked for more than 120 seconds. - [ 9184.452024] Tainted: P OE 4.8.0-32-generic #34-Ubuntu - [ 9184.452025] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. - [ 9184.452026] bitcoin-msghand D a23eeb23bd38 0 663 26994 0x - [ 9184.452028] a23eeb23bd38 00ffa23eab434000 a240ecf7 a24095148000 - [ 9184.452030] a23eeb23bd20 a23eeb23c000 a240aa940828 a240aa940800 - [ 9184.452031] a240aa940980 a23eeb23bd50 bbe96b15 - [ 9184.452033] Call Trace: - [ 9184.452034] [] schedule+0x35/0x80 - [ 9184.452039] [] cv_wait_common+0x110/0x130 [spl] - [ 9184.452041] [] ? wake_atomic_t_function+0x60/0x60 - [ 9184.452044] [] __cv_wait+0x15/0x20 [spl] - [ 9184.452074] [] zil_commit.part.11+0x79/0x7a0 [zfs] - [ 9184.452079] [] ? tsd_hash_search.isra.0+0x46/0xa0 [spl] - [ 9184.452083] [] ? tsd_set+0x2b4/0x500 [spl] - [ 9184.452084] [] ? mutex_lock+0x12/0x30 - [ 9184.452113] [] zil_commit+0x17/0x20 [zfs] - [ 9184.452141] [] zfs_fsync+0x7a/0xf0 [zfs] - [ 9184.452168] [] zpl_fsync+0x68/0xa0 [zfs] - [ 9184.452170] [] vfs_fsync_range+0x4b/0xb0 - [ 9184.452172] [] do_fsync+0x3d/0x70 - [ 9184.452173] [] SyS_fdatasync+0x13/0x20 - [ 9184.452174] [] entry_SYSCALL_64_fastpath+0x1e/0xa8 - [ 9305.221219] INFO: task txg_sync:11471 blocked for more than 120 seconds. - [
[Bug 1654517] Re: ZFS I/O hangs for minutes
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: zfs-linux (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654517 Title: ZFS I/O hangs for minutes To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1654517/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654517] Re: ZFS I/O hangs for minutes
** Package changed: linux (Ubuntu) => zfs-linux (Ubuntu) ** Changed in: zfs-linux (Ubuntu) Status: Confirmed => New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654517 Title: ZFS I/O hangs for minutes To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1654517/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1654517] Re: ZFS I/O hangs for minutes
apport information ** Tags added: apport-collected ** Description changed: I/O for multiple programs, like `thunderbird`, `firefox`, etc., hangs for minutes, `dmesg` contains [ 9184.451606] INFO: task txg_sync:11471 blocked for more than 120 seconds. [ 9184.451610] Tainted: P OE 4.8.0-32-generic #34-Ubuntu [ 9184.451611] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 9184.451612] txg_syncD a240ab3a7aa8 0 11471 2 0x [ 9184.451616] a240ab3a7aa8 00ffbb6ade1f a24095148000 a240e5ca5580 [ 9184.451618] 0046 a240ab3a8000 a240ff359200 7fff [ 9184.451620] a23d36cf9050 0001 a240ab3a7ac0 bbe96b15 [ 9184.451621] Call Trace: [ 9184.451627] [] schedule+0x35/0x80 [ 9184.451628] [] schedule_timeout+0x22a/0x3f0 [ 9184.451631] [] ? __switch_to+0x2ce/0x6c0 [ 9184.451633] [] ? pick_next_task_fair+0x48c/0x4c0 [ 9184.451635] [] ? ktime_get+0x41/0xb0 [ 9184.451636] [] io_schedule_timeout+0xa4/0x110 [ 9184.451644] [] cv_wait_common+0xb2/0x130 [spl] [ 9184.451646] [] ? wake_atomic_t_function+0x60/0x60 [ 9184.451650] [] __cv_wait_io+0x18/0x20 [spl] [ 9184.451689] [] zio_wait+0xfd/0x1d0 [zfs] [ 9184.451716] [] dsl_pool_sync+0xb8/0x480 [zfs] [ 9184.451745] [] spa_sync+0x37f/0xb30 [zfs] [ 9184.451747] [] ? default_wake_function+0x12/0x20 [ 9184.451779] [] txg_sync_thread+0x3a5/0x600 [zfs] [ 9184.451807] [] ? txg_delay+0x160/0x160 [zfs] [ 9184.451811] [] thread_generic_wrapper+0x71/0x80 [spl] [ 9184.451815] [] ? __thread_exit+0x20/0x20 [spl] [ 9184.451817] [] kthread+0xd8/0xf0 [ 9184.451819] [] ret_from_fork+0x1f/0x40 [ 9184.451821] [] ? kthread_create_on_node+0x1e0/0x1e0 [ 9184.451849] INFO: task mozStorage #2:21607 blocked for more than 120 seconds. [ 9184.451851] Tainted: P OE 4.8.0-32-generic #34-Ubuntu [ 9184.451852] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 9184.451853] mozStorage #2 D a23fe8a5bd38 0 21607 19750 0x0004 [ 9184.451855] a23fe8a5bd38 00ffa240ee8feb40 a240ecf72ac0 a2403803b900 [ 9184.451857] bc2c02f7 a23fe8a5c000 a240aa940828 a240aa940800 [ 9184.451858] a240aa940980 a23fe8a5bd50 bbe96b15 [ 9184.451860] Call Trace: [ 9184.451861] [] schedule+0x35/0x80 [ 9184.451866] [] cv_wait_common+0x110/0x130 [spl] [ 9184.451868] [] ? wake_atomic_t_function+0x60/0x60 [ 9184.451872] [] __cv_wait+0x15/0x20 [spl] [ 9184.451904] [] zil_commit.part.11+0x79/0x7a0 [zfs] [ 9184.451909] [] ? tsd_hash_search.isra.0+0x46/0xa0 [spl] [ 9184.451913] [] ? tsd_set+0x2b4/0x500 [spl] [ 9184.451914] [] ? mutex_lock+0x12/0x30 [ 9184.451945] [] zil_commit+0x17/0x20 [zfs] [ 9184.451975] [] zfs_fsync+0x7a/0xf0 [zfs] [ 9184.452005] [] zpl_fsync+0x68/0xa0 [zfs] [ 9184.452008] [] vfs_fsync_range+0x4b/0xb0 [ 9184.452010] [] do_fsync+0x3d/0x70 [ 9184.452011] [] SyS_fsync+0x10/0x20 [ 9184.452013] [] entry_SYSCALL_64_fastpath+0x1e/0xa8 [ 9184.452023] INFO: task bitcoin-msghand:663 blocked for more than 120 seconds. [ 9184.452024] Tainted: P OE 4.8.0-32-generic #34-Ubuntu [ 9184.452025] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 9184.452026] bitcoin-msghand D a23eeb23bd38 0 663 26994 0x [ 9184.452028] a23eeb23bd38 00ffa23eab434000 a240ecf7 a24095148000 [ 9184.452030] a23eeb23bd20 a23eeb23c000 a240aa940828 a240aa940800 [ 9184.452031] a240aa940980 a23eeb23bd50 bbe96b15 [ 9184.452033] Call Trace: [ 9184.452034] [] schedule+0x35/0x80 [ 9184.452039] [] cv_wait_common+0x110/0x130 [spl] [ 9184.452041] [] ? wake_atomic_t_function+0x60/0x60 [ 9184.452044] [] __cv_wait+0x15/0x20 [spl] [ 9184.452074] [] zil_commit.part.11+0x79/0x7a0 [zfs] [ 9184.452079] [] ? tsd_hash_search.isra.0+0x46/0xa0 [spl] [ 9184.452083] [] ? tsd_set+0x2b4/0x500 [spl] [ 9184.452084] [] ? mutex_lock+0x12/0x30 [ 9184.452113] [] zil_commit+0x17/0x20 [zfs] [ 9184.452141] [] zfs_fsync+0x7a/0xf0 [zfs] [ 9184.452168] [] zpl_fsync+0x68/0xa0 [zfs] [ 9184.452170] [] vfs_fsync_range+0x4b/0xb0 [ 9184.452172] [] do_fsync+0x3d/0x70 [ 9184.452173] [] SyS_fdatasync+0x13/0x20 [ 9184.452174] [] entry_SYSCALL_64_fastpath+0x1e/0xa8 [ 9305.221219] INFO: task txg_sync:11471 blocked for more than 120 seconds. [ 9305.221223] Tainted: P OE 4.8.0-32-generic
[Bug 1654517] Re: ZFS I/O hangs for minutes
** Package changed: zfs-linux (Ubuntu) => linux (Ubuntu) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1654517 Title: ZFS I/O hangs for minutes To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1654517/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
