[Bug 1862822] Re: init_on_alloc=1 causes big performance regression for ZFS

2020-02-12 Thread Matthew Ahrens
apport information

** Tags added: apport-collected bionic uec-images

** Description changed:

  Apologies if this is not the right forum to raise this concern.  If not
  please point me to a better place to have this discussion.
  
  The 5.3 linux kernel added a new feature which allows pages to be zeroed
  when allocating or freeing them: init_on_alloc and init_on_free. The
  init_on_alloc flag is enabled by default on the Ubuntu 18.04 HWE kernel.
  
  Linux kernel commit:
  
https://github.com/torvalds/linux/commit/6471384af2a6530696fc0203bafe4de41a23c9ef
  
  ZFS allocates and frees pages frequently (via the ABD structure), e.g.
  for every disk access. The additional overhead of zeroing these pages is
  significant. For example, I measured a ~40% regression in performance of
  an uncached "zfs send ... >/dev/null".  While this is a specific
  workload, this is a general problem because whenever we read from disk
  we are going to allocate pages, unnecessarily zero them, and then
  immediately fill them with the data from disk.
  
  I don't see any mechanism for ZFS to request that pages *not* be zeroed
  when they are allocated.
  
  This performance regression has been noted by users of ZFS on Ubuntu:
  https://github.com/zfsonlinux/zfs/issues/9910
  
  I'd like to request that the default in Ubuntu be changed to
  init_on_alloc=0, so that users of ZFS on Ubuntu do not experience this
  performance regression.
  
  (FYI, my employer, Delphix, has worked around this by changing the
  default in our product, but my concern is that the vast majority of
  Ubuntu users will simply experience this as "Ubuntu HWE is much slower
  than before".)
  
  $ cat /proc/version_signature
  Ubuntu 5.3.0-28.30~18.04.1-generic 5.3.13
+ --- 
+ ProblemType: Bug
+ AlsaDevices:
+  total 0
+  crw-rw 1 root audio 116,  1 Feb 11 18:34 seq
+  crw-rw 1 root audio 116, 33 Feb 11 18:34 timer
+ AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
+ ApportVersion: 2.20.9-0ubuntu7.9
+ Architecture: amd64
+ ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 
'arecord'
+ AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', 
'/dev/snd/timer'] failed with exit code 1:
+ DistroRelease: Ubuntu 18.04
+ IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
+ Lsusb: Error: [Errno 2] No such file or directory: 'lsusb': 'lsusb'
+ MachineType: VMware, Inc. VMware Virtual Platform
+ NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
+ Package: linux (not installed)
+ PciMultimedia:
+  
+ ProcEnviron:
+  TERM=xterm
+  PATH=(custom, no user)
+  LANG=en_US.UTF-8
+  SHELL=/bin/bash
+ ProcFB: 0 svgadrmfb
+ ProcKernelCmdLine: 
BOOT_IMAGE=/ROOT/username.90dy59N/root@/boot/vmlinuz-5.3.0-28-generic 
root=ZFS=rpool/ROOT/username.90dy59N/root ro console=tty0 console=ttyS0,38400n8 
mitigations=off ipv6.disable=1 elevator=noop crashkernel=256M,high 
crashkernel=256M,low init_on_alloc=0
+ ProcVersionSignature: Ubuntu 5.3.0-28.30~18.04.1-generic 5.3.13
+ RelatedPackageVersions:
+  linux-restricted-modules-5.3.0-28-generic N/A
+  linux-backports-modules-5.3.0-28-generic  N/A
+  linux-firmware1.173.14
+ RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
+ Tags:  bionic uec-images
+ Uname: Linux 5.3.0-28-generic x86_64
+ UnreportableReason: This report is about a package that is not installed.
+ UpgradeStatus: No upgrade log present (probably fresh install)
+ UserGroups: linux-staff root
+ WifiSyslog:
+  
+ _MarkForUpload: False
+ dmi.bios.date: 12/12/2018
+ dmi.bios.vendor: Phoenix Technologies LTD
+ dmi.bios.version: 6.00
+ dmi.board.name: 440BX Desktop Reference Platform
+ dmi.board.vendor: Intel Corporation
+ dmi.board.version: None
+ dmi.chassis.asset.tag: No Asset Tag
+ dmi.chassis.type: 1
+ dmi.chassis.vendor: No Enclosure
+ dmi.chassis.version: N/A
+ dmi.modalias: 
dmi:bvnPhoenixTechnologiesLTD:bvr6.00:bd12/12/2018:svnVMware,Inc.:pnVMwareVirtualPlatform:pvrNone:rvnIntelCorporation:rn440BXDesktopReferencePlatform:rvrNone:cvnNoEnclosure:ct1:cvrN/A:
+ dmi.product.name: VMware Virtual Platform
+ dmi.product.version: None
+ dmi.sys.vendor: VMware, Inc.

** Attachment added: "CRDA.txt"
   https://bugs.launchpad.net/bugs/1862822/+attachment/5327846/+files/CRDA.txt

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1862822

Title:
  init_on_alloc=1 causes big performance regression for ZFS

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1862822/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1862822] CurrentDmesg.txt

2020-02-12 Thread Matthew Ahrens
apport information

** Attachment added: "CurrentDmesg.txt"
   
https://bugs.launchpad.net/bugs/1862822/+attachment/5327847/+files/CurrentDmesg.txt

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1862822

Title:
  init_on_alloc=1 causes big performance regression for ZFS

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1862822/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1862822] ProcModules.txt

2020-02-12 Thread Matthew Ahrens
apport information

** Attachment added: "ProcModules.txt"
   
https://bugs.launchpad.net/bugs/1862822/+attachment/5327851/+files/ProcModules.txt

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1862822

Title:
  init_on_alloc=1 causes big performance regression for ZFS

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1862822/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1862822] UdevDb.txt

2020-02-12 Thread Matthew Ahrens
apport information

** Attachment added: "UdevDb.txt"
   https://bugs.launchpad.net/bugs/1862822/+attachment/5327852/+files/UdevDb.txt

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1862822

Title:
  init_on_alloc=1 causes big performance regression for ZFS

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1862822/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1862822] ProcCpuinfoMinimal.txt

2020-02-12 Thread Matthew Ahrens
apport information

** Attachment added: "ProcCpuinfoMinimal.txt"
   
https://bugs.launchpad.net/bugs/1862822/+attachment/5327849/+files/ProcCpuinfoMinimal.txt

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1862822

Title:
  init_on_alloc=1 causes big performance regression for ZFS

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1862822/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1862822] ProcInterrupts.txt

2020-02-12 Thread Matthew Ahrens
apport information

** Attachment added: "ProcInterrupts.txt"
   
https://bugs.launchpad.net/bugs/1862822/+attachment/5327850/+files/ProcInterrupts.txt

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1862822

Title:
  init_on_alloc=1 causes big performance regression for ZFS

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1862822/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1862822] Lspci.txt

2020-02-12 Thread Matthew Ahrens
apport information

** Attachment added: "Lspci.txt"
   https://bugs.launchpad.net/bugs/1862822/+attachment/5327848/+files/Lspci.txt

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1862822

Title:
  init_on_alloc=1 causes big performance regression for ZFS

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1862822/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1862822] Re: init_on_alloc=1 causes big performance regression for ZFS

2020-02-11 Thread Matthew Ahrens
I installed apport and python3-launchpadlib, and ran the requested
command:

$ apport-collect 1862822
The authorization page:
 
(https://launchpad.net/+authorize-token?oauth_token=KJm30XZW6qPv9HDzn45k&allow_permission=DESKTOP_INTEGRATION)
should be opening in your browser. Use your browser to authorize
this program to access Launchpad on your behalf.
Waiting to hear from Launchpad about your decision...

However, I don't have a web browser installed.  Let me know if there's
another way to gather the requested information.

** Changed in: linux (Ubuntu)
   Status: Incomplete => Confirmed

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1862822

Title:
  init_on_alloc=1 causes big performance regression for ZFS

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1862822/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1862822] [NEW] init_on_alloc=1 causes big performance regression for ZFS

2020-02-11 Thread Matthew Ahrens
Public bug reported:

Apologies if this is not the right forum to raise this concern.  If not
please point me to a better place to have this discussion.

The 5.3 linux kernel added a new feature which allows pages to be zeroed
when allocating or freeing them: init_on_alloc and init_on_free. The
init_on_alloc flag is enabled by default on the Ubuntu 18.04 HWE kernel.

Linux kernel commit:
https://github.com/torvalds/linux/commit/6471384af2a6530696fc0203bafe4de41a23c9ef

ZFS allocates and frees pages frequently (via the ABD structure), e.g.
for every disk access. The additional overhead of zeroing these pages is
significant. For example, I measured a ~40% regression in performance of
an uncached "zfs send ... >/dev/null".  While this is a specific
workload, this is a general problem because whenever we read from disk
we are going to allocate pages, unnecessarily zero them, and then
immediately fill them with the data from disk.

I don't see any mechanism for ZFS to request that pages *not* be zeroed
when they are allocated.

This performance regression has been noted by users of ZFS on Ubuntu:
https://github.com/zfsonlinux/zfs/issues/9910

I'd like to request that the default in Ubuntu be changed to
init_on_alloc=0, so that users of ZFS on Ubuntu do not experience this
performance regression.

(FYI, my employer, Delphix, has worked around this by changing the
default in our product, but my concern is that the vast majority of
Ubuntu users will simply experience this as "Ubuntu HWE is much slower
than before".)

$ cat /proc/version_signature
Ubuntu 5.3.0-28.30~18.04.1-generic 5.3.13

** Affects: linux (Ubuntu)
 Importance: Undecided
 Status: New

** Attachment added: "lspci-vnvn.log"
   
https://bugs.launchpad.net/bugs/1862822/+attachment/5327397/+files/lspci-vnvn.log

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1862822

Title:
  init_on_alloc=1 causes big performance regression for ZFS

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1862822/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1857398] Re: ubiquity should support encryption by default with zfsroot, with users able to opt in to running change-key after install

2020-01-10 Thread Matthew Ahrens
I agree with what Richard said above in comment #14, especially:

> The installer should prompt (with a checkbox) for whether the user
wants encryption. It should default to off. If the user selects the
checkbox, prompt them for a passphrase. Setup encryption using that
passphrase.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1857398

Title:
  ubiquity should support encryption by default with zfsroot, with users
  able to opt in to running change-key after install

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/1857398/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs