[Bug 1567557] Re: Performance degradation of "zfs clone"

2017-09-04 Thread Launchpad Bug Tracker
This bug was fixed in the package zfs-linux - 0.6.5.6-0ubuntu18 --- zfs-linux (0.6.5.6-0ubuntu18) xenial; urgency=medium * Improve cloning performance for large numbers of clones (LP: #1567557) Bump zcmd buffer from 16K to 256K. * Correctly install libraries such as

[Bug 1567557] Re: Performance degradation of "zfs clone"

2017-09-04 Thread Łukasz Zemczak
As per comment #24, considering the upload to be tested for zesty as well ^ ** Tags added: verification-done-zesty -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1567557 Title: Performance

[Bug 1567557] Re: Performance degradation of "zfs clone"

2017-09-04 Thread Launchpad Bug Tracker
This bug was fixed in the package zfs-linux - 0.6.5.9-5ubuntu4.2 --- zfs-linux (0.6.5.9-5ubuntu4.2) zesty; urgency=medium * Improve cloning performance for large numbers of clones (LP: #1567557) Bump zcmd buffer from 16K to 256K. -- Colin Ian King

[Bug 1567557] Re: Performance degradation of "zfs clone"

2017-08-31 Thread Colin Ian King
@Brian, Yes indeed I have performed exhaustive testing on this on Xenial, Zesty, Artful. A full exhaustive test takes 5 hours and I have performed the upgrade steps and checked this out. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1567557] Re: Performance degradation of "zfs clone"

2017-08-31 Thread Brian Murray
Has this also been verified with Ubuntu 17.04? While its unlikely people are upgrading from 16.04 to 17.04 it is an upgrade path and we should ensure this fix does not regress and release these updates together. -- You received this bug notification because you are a member of Ubuntu Bugs, which

[Bug 1567557] Re: Performance degradation of "zfs clone"

2017-08-22 Thread Stéphane Graber
** No longer affects: lxd (Ubuntu Xenial) ** No longer affects: lxd (Ubuntu Zesty) ** No longer affects: lxd (Ubuntu Artful) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1567557 Title:

[Bug 1567557] Re: Performance degradation of "zfs clone"

2017-08-22 Thread Stéphane Graber
** No longer affects: lxd (Ubuntu) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1567557 Title: Performance degradation of "zfs clone" To manage notifications about this bug go to:

[Bug 1567557] Re: Performance degradation of "zfs clone"

2017-08-18 Thread Colin Ian King
Tested with zfs 0.6.5.6-0ubuntu18 on Xenial. Passes zfs smoke tests. The performance increase on Xenial is not so significant than later releases, but I'm seeing double digit % increase in performance with a test of 3000 clones. ** Tags added: verification-done-xenial -- You received this bug

[Bug 1567557] Re: Performance degradation of "zfs clone"

2017-08-17 Thread Brian Murray
Hello Stéphane, or anyone else affected, Accepted zfs-linux into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/zfs- linux/0.6.5.6-0ubuntu18 in a few hours, and then in the -proposed repository. Please help us by testing this new package.

[Bug 1567557] Re: Performance degradation of "zfs clone"

2017-08-10 Thread Brian Murray
Hello Stéphane, or anyone else affected, Accepted zfs-linux into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/zfs- linux/0.6.5.9-5ubuntu4.2 in a few hours, and then in the -proposed repository. Please help us by testing this new package.

[Bug 1567557] Re: Performance degradation of "zfs clone"

2017-08-10 Thread Colin Ian King
** Description changed: + [SRU Justification] + + Creating tens of hundreds of clones can be prohibitively slow. The + underlying mechanism to gather clone information is using a 16K buffer + which limits performance. Also, the initial assumption is to pass in + zero sized buffer to the

[Bug 1567557] Re: Performance degradation of "zfs clone"

2017-08-10 Thread Colin Ian King
** Changed in: zfs-linux (Ubuntu Xenial) Assignee: (unassigned) => Colin Ian King (colin-king) ** Changed in: zfs-linux (Ubuntu Zesty) Assignee: (unassigned) => Colin Ian King (colin-king) ** Changed in: zfs-linux (Ubuntu Zesty) Status: New => In Progress ** Changed in:

[Bug 1567557] Re: Performance degradation of "zfs clone"

2017-08-08 Thread Launchpad Bug Tracker
This bug was fixed in the package zfs-linux - 0.6.5.11-1ubuntu2 --- zfs-linux (0.6.5.11-1ubuntu2) artful; urgency=medium * Improve cloning performance for large numbers of clones (LP: #1567557) Bump zcmd buffer from 16K to 256K. -- Colin Ian King

[Bug 1567557] Re: Performance degradation of "zfs clone"

2017-08-08 Thread Colin Ian King
I've bumped up the zcmd size to reduce the double ioctl() calls and we're seeing 25-40% improvement clones > 1000. This is an interim fix until we get a better fix from upstream. I'm going to SRU this once we're OK with it in Arful. Attached are the new benchmarks. ** Attachment added:

[Bug 1567557] Re: Performance degradation of "zfs clone"

2017-08-08 Thread Colin Ian King
** Also affects: zfs-linux (Ubuntu Artful) Importance: Low Assignee: Colin Ian King (colin-king) Status: Triaged ** Also affects: lxd (Ubuntu Artful) Importance: High Status: Confirmed ** Also affects: zfs-linux (Ubuntu Xenial) Importance: Undecided Status: New

[Bug 1567557] Re: Performance degradation of "zfs clone"

2017-07-20 Thread Colin Ian King
** Summary changed: - Performance degradation of "zfs clone" when under load + Performance degradation of "zfs clone" -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1567557 Title: Performance

[Bug 1567557] Re: Performance degradation of "zfs clone" when under load

2017-07-19 Thread Bug Watch Updater
** Changed in: zfs Status: Unknown => New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1567557 Title: Performance degradation of "zfs clone" when under load To manage notifications about

[Bug 1567557] Re: Performance degradation of "zfs clone" when under load

2017-07-19 Thread Colin Ian King
Based on measurements, we get a x^3 polynomial in clone time. Sub 2000 clones it's not too bad, but over 2000 clones the delay in the ioctl() ramps up pretty quickly. I've got a fairly solid set of stats and drew some estimates based on the trend line. See attached datasheet. ** Attachment

[Bug 1567557] Re: Performance degradation of "zfs clone" when under load

2017-07-19 Thread Colin Ian King
Running strace against zfs create I see the following ioctl() taking the time: 1500475028.118005 ioctl(3, _IOC(0, 0x5a, 0x12, 0x00), 0x7ffc7c2184f0) = -1 ENOMEM (Cannot allocate memory) <0.390093> 1500475028.508153 mmap(NULL, 290816, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =

[Bug 1567557] Re: Performance degradation of "zfs clone" when under load

2017-07-05 Thread Colin Ian King
So compound that with the huge amount of CPU suckage I'm seeing with lxd... -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1567557 Title: Performance degradation of "zfs clone" when under load To

[Bug 1567557] Re: Performance degradation of "zfs clone" when under load

2017-07-05 Thread Stéphane Graber
(but doing something pretty similar to what LXD does internally as far as clones and mountpoint handling) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1567557 Title: Performance degradation of

[Bug 1567557] Re: Performance degradation of "zfs clone" when under load

2017-07-05 Thread Stéphane Graber
I'm seeing similar performance degradation on xenial except that the server I used seems to be pretty slow to start with too: root@athos:~# sh test.sh Creating 100 clones Took: 11 seconds (9/s) Creating 200 clones Took: 46 seconds (4/s) Creating 400 clones Took: 297 seconds (1/s) -- You

[Bug 1567557] Re: Performance degradation of "zfs clone" when under load

2017-07-05 Thread Stéphane Graber
That's pure ZFS completely outside of LXD. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1567557 Title: Performance degradation of "zfs clone" when under load To manage notifications about this

[Bug 1567557] Re: Performance degradation of "zfs clone" when under load

2017-07-05 Thread Stéphane Graber
Creating 100 clones Took: 4 seconds (25/s) Creating 200 clones Took: 13 seconds (15/s) Creating 400 clones Took: 46 seconds (8/s) Creating 600 clones Took: 156 seconds (3/s) ``` #!/bin/sh zfs destroy -R castiana/testzfs rm -Rf /tmp/testzfs zfs create castiana/testzfs -o mountpoint=none zfs

[Bug 1567557] Re: Performance degradation of "zfs clone" when under load

2017-07-05 Thread Colin Ian King
Looks like there is a lot of contention on a futex and the underlying database too. Perf record and perf report can show that most of the issues are more to do with lxd, database and futex lock contention. Once these are resolved, I'll be happy to re-analyze the entire ZFS + lxd stack, but I

[Bug 1567557] Re: Performance degradation of "zfs clone" when under load

2017-07-05 Thread Colin Ian King
Running health-check against lxd I see that there is a lot of contention on a futex and a load of context switches going on in lxd. ** Attachment added: "output from health-check when examining lxd"

[Bug 1567557] Re: Performance degradation of "zfs clone" when under load

2017-07-05 Thread Colin Ian King
I've now compared ZFS with BTRFS and the overall CPU load profile from lxd in both scenarios is surprisingly similar - lxd is eating a lot of CPU and I think there is an underlying issue there rather than a massively broken performance issue with ZFS. See the attached data sheet; the CPU

[Bug 1567557] Re: Performance degradation of "zfs clone" when under load

2017-07-05 Thread Colin Ian King
After a load tests, I've found the same anomaly with btrfs. I'm using a 80GB btrfs raw device, 16GB memory, 16 CPU threads: BTRFS: Time 24.02632 54.58464 125.520 128 272.835 256 809.038 512 3451.172 1024 So the pinch point happens later than with ZFS, but there seems to

[Bug 1567557] Re: Performance degradation of "zfs clone" when under load

2017-07-05 Thread Colin Ian King
Most of the CPU being consumed is in strcmp() - 99.39% 0.00% zfs[kernel.kallsyms] [k] zfsdev_ioctl ▒ - 99.39% zfsdev_ioctl

[Bug 1567557] Re: Performance degradation of "zfs clone" when under load

2017-07-05 Thread Colin Ian King
I've created 4096 zfs clones using the same commands that lxd is using and I have observed that the biggest pinch point is when fetching the zfs clone stats as this has to lock using dsl_dataset_hold_obj, fetch the data and then unlock with dsl_dataset_rele. Traversing hundreds of clones is

[Bug 1567557] Re: Performance degradation of "zfs clone" when under load

2017-05-31 Thread Stéphane Graber
Our test machines aren't particularly impressive, just 12GB of RAM or so. Note that as can be seen above, we're using Alpine (busybox) images rather than Ubuntu to limit the resource usage and get us to a lot more containers per system. -- You received this bug notification because you are a

[Bug 1567557] Re: Performance degradation of "zfs clone" when under load

2017-05-31 Thread Stéphane Graber
I'm trying to remember if we had to bump any of the sysctls to actually reach 1024 containers, I don't think any of the usual suspects would be in play until you reach 2000+ Alpine containers though. If you do run out of some kernel resources, you can try applying the following sysctls to get

[Bug 1567557] Re: Performance degradation of "zfs clone" when under load

2017-05-31 Thread Colin Ian King
I've tried to reproduce this but when I pass more than 256 containers I run out of resources. What kind of configuration are you using to test this. Once I can reproduce this I'll try and see what the performance constraint is. ** Changed in: zfs-linux (Ubuntu) Status: In Progress =>

[Bug 1567557] Re: Performance degradation of "zfs clone" when under load

2017-04-20 Thread Colin Ian King
** Changed in: zfs-linux (Ubuntu) Importance: Undecided => Medium ** Changed in: zfs-linux (Ubuntu) Assignee: (unassigned) => Colin Ian King (colin-king) ** Changed in: zfs-linux (Ubuntu) Status: New => In Progress -- You received this bug notification because you are a member

[Bug 1567557] Re: Performance degradation of "zfs clone" when under load

2016-04-25 Thread Richard Laager
Since you've already filed one bug report upstream, would you be interested in filing this one upstream? I can certainly copy-and-paste it upstream, but it seems like it'd be better to have it come from you. I don't know anything about LXD. (I'm just trying to help out with ZoL bug reports.) --