Looks like there is a lot of contention on a futex and the underlying database too. Perf record and perf report can show that most of the issues are more to do with lxd, database and futex lock contention. Once these are resolved, I'll be happy to re-analyze the entire ZFS + lxd stack, but I believe the main issue is lxd eating cycles and not ZFS per se.
- 27.21% 1.05% lxd [kernel.kallsyms]
[k] entry_SYSCALL_64_fastpath
- 26.16% entry_SYSCALL_64_fastpath
- 19.28% sys_futex
- 18.91% do_futex
- 12.70% futex_wake
+ 9.91% wake_up_q
+ 0.90% _raw_spin_lock
+ 5.00% futex_wait
+ 1.48% sys_select
+ 0.73% sys_newstat
+ 0.57% syscall_return_slowpath
+ 0.54% sys_pread64
- 25.16% 0.00% lxd libgolang-1.8-std.so.1
[.] runtime.goexit ▒
- runtime.goexit
- 14.98% main.(*operation).Run.func1
- 13.92% main.createFromImage.func1
- 13.86% main.containerCreateFromImage
- 12.26% main.containerCreateInternal
- 12.24% main.containerLXCCreate
- 11.30% main.networkUpdateStatic
- 9.16% main.containerLoadByName
- 4.79% main.dbContainerGet
+ 2.15% main.dbContainerConfig
+ 1.13% main.dbQueryRowScan
+ 0.82% main.dbContainerProfiles
+ 0.59% main.dbDevices
+ 4.33% main.containerLXCLoad
+ 1.98% main.(*containerLXC).fillNetworkDevice
+ 0.86% main.(*containerLXC).ConfigKeySet
+ 0.70% main.dbImageGet
0.67% main.(*storageZfs).ContainerCreateFromImage
+ 1.05% main.containerStatePut.func1
+ 2.37% main.deviceEventListener
+ 2.21% runtime.mcall
+ 1.70% net/http.(*conn).serve
+ 1.27% database/sql.(*Rows).awaitDone
+ 0.82% runtime.systemstack
+ 19.33% 0.33% lxd [kernel.kallsyms]
[k] sys_futex
+ 18.96% 0.43% lxd [kernel.kallsyms]
[k] do_futex
- 14.98% 0.00% lxd lxd
[.] main.(*operation).Run.func1 ▒
- main.(*operation).Run.func1
- 13.92% main.createFromImage.func1
- 13.86% main.containerCreateFromImage
- 12.26% main.containerCreateInternal
- 12.24% main.containerLXCCreate
- 11.30% main.networkUpdateStatic
- 9.16% main.containerLoadByName
- 4.79% main.dbContainerGet
+ 2.15% main.dbContainerConfig
+ 1.13% main.dbQueryRowScan
+ 0.82% main.dbContainerProfiles
+ 0.59% main.dbDevices
** Changed in: lxd (Ubuntu)
Importance: Undecided => High
** Changed in: zfs-linux (Ubuntu)
Importance: Medium => Low
** Changed in: lxd (Ubuntu)
Status: New => Confirmed
** Changed in: zfs-linux (Ubuntu)
Status: In Progress => Confirmed
** Changed in: zfs-linux (Ubuntu)
Status: Confirmed => Triaged
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1567557
Title:
Performance degradation of "zfs clone" when under load
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/lxd/+bug/1567557/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
