[Kernel-packages] [Bug 1725859] Re: zfs frequently hangs (up to 30 secs) during sequential read
Hello Colin, I set a couple of ZFS options a day or two ago, and have bee playing videos to see if we still have the problem. The options I set were: o zfs set xattr=sa storage o zfs set xattr=sa storage/vault o zfs set ralatime=on storage o zfs set ralatime=on storage/vault We've played about four hours of video, and we have not seen the hang. Do you still want to pursue this? I could set the options back and do the trace or leave the options and do the trace, or should we just consider this fixed? My only concern about the latter option is that the installation was a default install with Ubuntu 17.10 and it seems these options should be the default on any linux distro. Let me know what you want to do? Best regards, Colin. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to zfs-linux in Ubuntu. https://bugs.launchpad.net/bugs/1725859 Title: zfs frequently hangs (up to 30 secs) during sequential read Status in zfs-linux package in Ubuntu: Incomplete Bug description: Updated to artful (17.10) yesterday. Trying to read (play video) from mirrored ZFS disks from an external USB 3 enclosure. Zpool is defined as: root@hetty:/home/crlb# zpool status pool: storage state: ONLINE scan: resilvered 20K in 0h0m with 0 errors on Fri Oct 20 18:38:49 2017 config: NAMESTATE READ WRITE CKSUM storage ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 sdb ONLINE 0 0 0 sdc ONLINE 0 0 0 errors: No known data errors root@hetty:/home/crlb# Found that I could re-create the problem with: rsync -av --progress Also found that: dd if=/dev/sdX of=/dev/null status=progress bs=1024 count=1000 Where "X" is either "b" or "c" does not hang. Installed: root@hetty:/home/crlb# apt list --installed | grep -i zfs libzfs2linux/artful,now 0.6.5.11-1ubuntu3 amd64 [installed,automatic] zfs-zed/artful,now 0.6.5.11-1ubuntu3 amd64 [installed,automatic] zfsutils-linux/artful,now 0.6.5.11-1ubuntu3 amd64 [installed] root@hetty:/home/crlb# Help please. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1725859/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1725859] Re: zfs frequently hangs (up to 30 secs) during sequential read
root@hetty:/home/crlb# blockdev --getpbsz /dev/sdb 512 root@hetty:/home/crlb# blockdev --getpbsz /dev/sdc 512 root@hetty:/home/crlb# -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to zfs-linux in Ubuntu. https://bugs.launchpad.net/bugs/1725859 Title: zfs frequently hangs (up to 30 secs) during sequential read Status in zfs-linux package in Ubuntu: Incomplete Bug description: Updated to artful (17.10) yesterday. Trying to read (play video) from mirrored ZFS disks from an external USB 3 enclosure. Zpool is defined as: root@hetty:/home/crlb# zpool status pool: storage state: ONLINE scan: resilvered 20K in 0h0m with 0 errors on Fri Oct 20 18:38:49 2017 config: NAMESTATE READ WRITE CKSUM storage ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 sdb ONLINE 0 0 0 sdc ONLINE 0 0 0 errors: No known data errors root@hetty:/home/crlb# Found that I could re-create the problem with: rsync -av --progress Also found that: dd if=/dev/sdX of=/dev/null status=progress bs=1024 count=1000 Where "X" is either "b" or "c" does not hang. Installed: root@hetty:/home/crlb# apt list --installed | grep -i zfs libzfs2linux/artful,now 0.6.5.11-1ubuntu3 amd64 [installed,automatic] zfs-zed/artful,now 0.6.5.11-1ubuntu3 amd64 [installed,automatic] zfsutils-linux/artful,now 0.6.5.11-1ubuntu3 amd64 [installed] root@hetty:/home/crlb# Help please. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1725859/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1725859] Re: zfs frequently hangs (up to 30 secs) during sequential read
Hello Colin, I have a couple more screen shots showing the raw performance of both drives; the first using 512MiB as you suggested and the second with a larger sample and using the dd commands status=progress option to monitor for pauses. Also in the second ss, I performed the test on an iso file from the zfs file system. I did not see any pauses on the raw devices, but then I only saw momentary pauses (one second or less) on the zfs file. Regards, Colin. ** Attachment added: "raw device performance" https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1725859/+attachment/5114647/+files/dd-1.png -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to zfs-linux in Ubuntu. https://bugs.launchpad.net/bugs/1725859 Title: zfs frequently hangs (up to 30 secs) during sequential read Status in zfs-linux package in Ubuntu: Incomplete Bug description: Updated to artful (17.10) yesterday. Trying to read (play video) from mirrored ZFS disks from an external USB 3 enclosure. Zpool is defined as: root@hetty:/home/crlb# zpool status pool: storage state: ONLINE scan: resilvered 20K in 0h0m with 0 errors on Fri Oct 20 18:38:49 2017 config: NAMESTATE READ WRITE CKSUM storage ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 sdb ONLINE 0 0 0 sdc ONLINE 0 0 0 errors: No known data errors root@hetty:/home/crlb# Found that I could re-create the problem with: rsync -av --progress Also found that: dd if=/dev/sdX of=/dev/null status=progress bs=1024 count=1000 Where "X" is either "b" or "c" does not hang. Installed: root@hetty:/home/crlb# apt list --installed | grep -i zfs libzfs2linux/artful,now 0.6.5.11-1ubuntu3 amd64 [installed,automatic] zfs-zed/artful,now 0.6.5.11-1ubuntu3 amd64 [installed,automatic] zfsutils-linux/artful,now 0.6.5.11-1ubuntu3 amd64 [installed] root@hetty:/home/crlb# Help please. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1725859/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1725859] Re: zfs frequently hangs (up to 30 secs) during sequential read
** Attachment added: "raw device and zfs performance on a larger sample." https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1725859/+attachment/5114648/+files/dd-2.png -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to zfs-linux in Ubuntu. https://bugs.launchpad.net/bugs/1725859 Title: zfs frequently hangs (up to 30 secs) during sequential read Status in zfs-linux package in Ubuntu: Incomplete Bug description: Updated to artful (17.10) yesterday. Trying to read (play video) from mirrored ZFS disks from an external USB 3 enclosure. Zpool is defined as: root@hetty:/home/crlb# zpool status pool: storage state: ONLINE scan: resilvered 20K in 0h0m with 0 errors on Fri Oct 20 18:38:49 2017 config: NAMESTATE READ WRITE CKSUM storage ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 sdb ONLINE 0 0 0 sdc ONLINE 0 0 0 errors: No known data errors root@hetty:/home/crlb# Found that I could re-create the problem with: rsync -av --progress Also found that: dd if=/dev/sdX of=/dev/null status=progress bs=1024 count=1000 Where "X" is either "b" or "c" does not hang. Installed: root@hetty:/home/crlb# apt list --installed | grep -i zfs libzfs2linux/artful,now 0.6.5.11-1ubuntu3 amd64 [installed,automatic] zfs-zed/artful,now 0.6.5.11-1ubuntu3 amd64 [installed,automatic] zfsutils-linux/artful,now 0.6.5.11-1ubuntu3 amd64 [installed] root@hetty:/home/crlb# Help please. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1725859/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1725859] Re: zfs frequently hangs (up to 30 secs) during sequential read
Hello Colin, Thank you for responding. To Answer your questions: 1. I have two external drives, sdb and sdc, in USB 3 Orico enclosure. The drives are mirrored. 2. Atom D525 with 4GiB RAM. The attached screenshot shows "vmstat 5 5" output while VLC is playing a movie. It shows 1.5GiB of free memory, no swap used. 3. Screenshot also shows output from "iostat -mx", and that I manually set the spl_kmem_cache_slab_limit to zero as suggested. I played a movie which paused repeatedly for 30 seconds and one time for several minutes. When the pause occurs, VLC is completely hung until the requested I/O completes. The end of pause is heralded by the I/O activity LED on the drive enclosure. I want to emphasize/re-iterate a couple of points from my original post: 1. This is not a VLC problem. I can recreate the problem with any app (cp, rsyn, etc) that reads sequentially from the ZFS file system. 2. I do not believe this is a drive/latency problem: I can dd the content of sdb and sdc (to /dev/null) without any hang. Regards, Colin. ** Attachment added: "spl_kmem_cache_slab_limit, vmstat, and iostat -mx" https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1725859/+attachment/5113194/+files/spl.png -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to zfs-linux in Ubuntu. https://bugs.launchpad.net/bugs/1725859 Title: zfs frequently hangs (up to 30 secs) during sequential read Status in zfs-linux package in Ubuntu: Incomplete Bug description: Updated to artful (17.10) yesterday. Trying to read (play video) from mirrored ZFS disks from an external USB 3 enclosure. Zpool is defined as: root@hetty:/home/crlb# zpool status pool: storage state: ONLINE scan: resilvered 20K in 0h0m with 0 errors on Fri Oct 20 18:38:49 2017 config: NAMESTATE READ WRITE CKSUM storage ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 sdb ONLINE 0 0 0 sdc ONLINE 0 0 0 errors: No known data errors root@hetty:/home/crlb# Found that I could re-create the problem with: rsync -av --progress Also found that: dd if=/dev/sdX of=/dev/null status=progress bs=1024 count=1000 Where "X" is either "b" or "c" does not hang. Installed: root@hetty:/home/crlb# apt list --installed | grep -i zfs libzfs2linux/artful,now 0.6.5.11-1ubuntu3 amd64 [installed,automatic] zfs-zed/artful,now 0.6.5.11-1ubuntu3 amd64 [installed,automatic] zfsutils-linux/artful,now 0.6.5.11-1ubuntu3 amd64 [installed] root@hetty:/home/crlb# Help please. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1725859/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1725859] Re: zfs frequently hangs (up to 30 secs) during sequential read
Just updated to: root@hetty:~# apt list --installed | awk '/zpool|zfs/' libzfs2linux/artful-updates,now 0.6.5.11-1ubuntu3.2 amd64 [installed,automatic] libzpool2linux/artful-updates,now 0.6.5.11-1ubuntu3.2 amd64 [installed] zfs-zed/artful-updates,now 0.6.5.11-1ubuntu3.2 amd64 [installed,automatic] zfsutils-linux/artful-updates,now 0.6.5.11-1ubuntu3.2 amd64 [installed] root@hetty:~# A little better, but the problem still happens. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to zfs-linux in Ubuntu. https://bugs.launchpad.net/bugs/1725859 Title: zfs frequently hangs (up to 30 secs) during sequential read Status in zfs-linux package in Ubuntu: New Bug description: Updated to artful (17.10) yesterday. Trying to read (play video) from mirrored ZFS disks from an external USB 3 enclosure. Zpool is defined as: root@hetty:/home/crlb# zpool status pool: storage state: ONLINE scan: resilvered 20K in 0h0m with 0 errors on Fri Oct 20 18:38:49 2017 config: NAMESTATE READ WRITE CKSUM storage ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 sdb ONLINE 0 0 0 sdc ONLINE 0 0 0 errors: No known data errors root@hetty:/home/crlb# Found that I could re-create the problem with: rsync -av --progress Also found that: dd if=/dev/sdX of=/dev/null status=progress bs=1024 count=1000 Where "X" is either "b" or "c" does not hang. Installed: root@hetty:/home/crlb# apt list --installed | grep -i zfs libzfs2linux/artful,now 0.6.5.11-1ubuntu3 amd64 [installed,automatic] zfs-zed/artful,now 0.6.5.11-1ubuntu3 amd64 [installed,automatic] zfsutils-linux/artful,now 0.6.5.11-1ubuntu3 amd64 [installed] root@hetty:/home/crlb# Help please. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1725859/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1725859] [NEW] zfs frequently hangs (up to 30 secs) during sequential read
Public bug reported: Updated to artful (17.10) yesterday. Trying to read (play video) from mirrored ZFS disks from an external USB 3 enclosure. Zpool is defined as: root@hetty:/home/crlb# zpool status pool: storage state: ONLINE scan: resilvered 20K in 0h0m with 0 errors on Fri Oct 20 18:38:49 2017 config: NAMESTATE READ WRITE CKSUM storage ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 sdb ONLINE 0 0 0 sdc ONLINE 0 0 0 errors: No known data errors root@hetty:/home/crlb# Found that I could re-create the problem with: rsync -av --progress Also found that: dd if=/dev/sdX of=/dev/null status=progress bs=1024 count=1000 Where "X" is either "b" or "c" does not hang. Installed: root@hetty:/home/crlb# apt list --installed | grep -i zfs libzfs2linux/artful,now 0.6.5.11-1ubuntu3 amd64 [installed,automatic] zfs-zed/artful,now 0.6.5.11-1ubuntu3 amd64 [installed,automatic] zfsutils-linux/artful,now 0.6.5.11-1ubuntu3 amd64 [installed] root@hetty:/home/crlb# Help please. ** Affects: zfs-linux (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to zfs-linux in Ubuntu. https://bugs.launchpad.net/bugs/1725859 Title: zfs frequently hangs (up to 30 secs) during sequential read Status in zfs-linux package in Ubuntu: New Bug description: Updated to artful (17.10) yesterday. Trying to read (play video) from mirrored ZFS disks from an external USB 3 enclosure. Zpool is defined as: root@hetty:/home/crlb# zpool status pool: storage state: ONLINE scan: resilvered 20K in 0h0m with 0 errors on Fri Oct 20 18:38:49 2017 config: NAMESTATE READ WRITE CKSUM storage ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 sdb ONLINE 0 0 0 sdc ONLINE 0 0 0 errors: No known data errors root@hetty:/home/crlb# Found that I could re-create the problem with: rsync -av --progress Also found that: dd if=/dev/sdX of=/dev/null status=progress bs=1024 count=1000 Where "X" is either "b" or "c" does not hang. Installed: root@hetty:/home/crlb# apt list --installed | grep -i zfs libzfs2linux/artful,now 0.6.5.11-1ubuntu3 amd64 [installed,automatic] zfs-zed/artful,now 0.6.5.11-1ubuntu3 amd64 [installed,automatic] zfsutils-linux/artful,now 0.6.5.11-1ubuntu3 amd64 [installed] root@hetty:/home/crlb# Help please. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1725859/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
