[Bug 1862822] Re: init_on_alloc=1 causes big performance regression for ZFS
apport information ** Tags added: apport-collected bionic uec-images ** Description changed: Apologies if this is not the right forum to raise this concern. If not please point me to a better place to have this discussion. The 5.3 linux kernel added a new feature which allows pages to be zeroed when allocating or freeing them: init_on_alloc and init_on_free. The init_on_alloc flag is enabled by default on the Ubuntu 18.04 HWE kernel. Linux kernel commit: https://github.com/torvalds/linux/commit/6471384af2a6530696fc0203bafe4de41a23c9ef ZFS allocates and frees pages frequently (via the ABD structure), e.g. for every disk access. The additional overhead of zeroing these pages is significant. For example, I measured a ~40% regression in performance of an uncached "zfs send ... >/dev/null". While this is a specific workload, this is a general problem because whenever we read from disk we are going to allocate pages, unnecessarily zero them, and then immediately fill them with the data from disk. I don't see any mechanism for ZFS to request that pages *not* be zeroed when they are allocated. This performance regression has been noted by users of ZFS on Ubuntu: https://github.com/zfsonlinux/zfs/issues/9910 I'd like to request that the default in Ubuntu be changed to init_on_alloc=0, so that users of ZFS on Ubuntu do not experience this performance regression. (FYI, my employer, Delphix, has worked around this by changing the default in our product, but my concern is that the vast majority of Ubuntu users will simply experience this as "Ubuntu HWE is much slower than before".) $ cat /proc/version_signature Ubuntu 5.3.0-28.30~18.04.1-generic 5.3.13 + --- + ProblemType: Bug + AlsaDevices: + total 0 + crw-rw 1 root audio 116, 1 Feb 11 18:34 seq + crw-rw 1 root audio 116, 33 Feb 11 18:34 timer + AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay' + ApportVersion: 2.20.9-0ubuntu7.9 + Architecture: amd64 + ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord' + AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: + DistroRelease: Ubuntu 18.04 + IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig' + Lsusb: Error: [Errno 2] No such file or directory: 'lsusb': 'lsusb' + MachineType: VMware, Inc. VMware Virtual Platform + NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair + Package: linux (not installed) + PciMultimedia: + + ProcEnviron: + TERM=xterm + PATH=(custom, no user) + LANG=en_US.UTF-8 + SHELL=/bin/bash + ProcFB: 0 svgadrmfb + ProcKernelCmdLine: BOOT_IMAGE=/ROOT/username.90dy59N/root@/boot/vmlinuz-5.3.0-28-generic root=ZFS=rpool/ROOT/username.90dy59N/root ro console=tty0 console=ttyS0,38400n8 mitigations=off ipv6.disable=1 elevator=noop crashkernel=256M,high crashkernel=256M,low init_on_alloc=0 + ProcVersionSignature: Ubuntu 5.3.0-28.30~18.04.1-generic 5.3.13 + RelatedPackageVersions: + linux-restricted-modules-5.3.0-28-generic N/A + linux-backports-modules-5.3.0-28-generic N/A + linux-firmware1.173.14 + RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill' + Tags: bionic uec-images + Uname: Linux 5.3.0-28-generic x86_64 + UnreportableReason: This report is about a package that is not installed. + UpgradeStatus: No upgrade log present (probably fresh install) + UserGroups: linux-staff root + WifiSyslog: + + _MarkForUpload: False + dmi.bios.date: 12/12/2018 + dmi.bios.vendor: Phoenix Technologies LTD + dmi.bios.version: 6.00 + dmi.board.name: 440BX Desktop Reference Platform + dmi.board.vendor: Intel Corporation + dmi.board.version: None + dmi.chassis.asset.tag: No Asset Tag + dmi.chassis.type: 1 + dmi.chassis.vendor: No Enclosure + dmi.chassis.version: N/A + dmi.modalias: dmi:bvnPhoenixTechnologiesLTD:bvr6.00:bd12/12/2018:svnVMware,Inc.:pnVMwareVirtualPlatform:pvrNone:rvnIntelCorporation:rn440BXDesktopReferencePlatform:rvrNone:cvnNoEnclosure:ct1:cvrN/A: + dmi.product.name: VMware Virtual Platform + dmi.product.version: None + dmi.sys.vendor: VMware, Inc. ** Attachment added: "CRDA.txt" https://bugs.launchpad.net/bugs/1862822/+attachment/5327846/+files/CRDA.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1862822 Title: init_on_alloc=1 causes big performance regression for ZFS To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1862822/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1862822] CurrentDmesg.txt
apport information ** Attachment added: "CurrentDmesg.txt" https://bugs.launchpad.net/bugs/1862822/+attachment/5327847/+files/CurrentDmesg.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1862822 Title: init_on_alloc=1 causes big performance regression for ZFS To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1862822/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1862822] ProcModules.txt
apport information ** Attachment added: "ProcModules.txt" https://bugs.launchpad.net/bugs/1862822/+attachment/5327851/+files/ProcModules.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1862822 Title: init_on_alloc=1 causes big performance regression for ZFS To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1862822/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1862822] UdevDb.txt
apport information ** Attachment added: "UdevDb.txt" https://bugs.launchpad.net/bugs/1862822/+attachment/5327852/+files/UdevDb.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1862822 Title: init_on_alloc=1 causes big performance regression for ZFS To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1862822/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1862822] ProcCpuinfoMinimal.txt
apport information ** Attachment added: "ProcCpuinfoMinimal.txt" https://bugs.launchpad.net/bugs/1862822/+attachment/5327849/+files/ProcCpuinfoMinimal.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1862822 Title: init_on_alloc=1 causes big performance regression for ZFS To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1862822/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1862822] ProcInterrupts.txt
apport information ** Attachment added: "ProcInterrupts.txt" https://bugs.launchpad.net/bugs/1862822/+attachment/5327850/+files/ProcInterrupts.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1862822 Title: init_on_alloc=1 causes big performance regression for ZFS To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1862822/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1862822] Lspci.txt
apport information ** Attachment added: "Lspci.txt" https://bugs.launchpad.net/bugs/1862822/+attachment/5327848/+files/Lspci.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1862822 Title: init_on_alloc=1 causes big performance regression for ZFS To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1862822/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1862822] Re: init_on_alloc=1 causes big performance regression for ZFS
I installed apport and python3-launchpadlib, and ran the requested command: $ apport-collect 1862822 The authorization page: (https://launchpad.net/+authorize-token?oauth_token=KJm30XZW6qPv9HDzn45k&allow_permission=DESKTOP_INTEGRATION) should be opening in your browser. Use your browser to authorize this program to access Launchpad on your behalf. Waiting to hear from Launchpad about your decision... However, I don't have a web browser installed. Let me know if there's another way to gather the requested information. ** Changed in: linux (Ubuntu) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1862822 Title: init_on_alloc=1 causes big performance regression for ZFS To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1862822/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1862822] [NEW] init_on_alloc=1 causes big performance regression for ZFS
Public bug reported: Apologies if this is not the right forum to raise this concern. If not please point me to a better place to have this discussion. The 5.3 linux kernel added a new feature which allows pages to be zeroed when allocating or freeing them: init_on_alloc and init_on_free. The init_on_alloc flag is enabled by default on the Ubuntu 18.04 HWE kernel. Linux kernel commit: https://github.com/torvalds/linux/commit/6471384af2a6530696fc0203bafe4de41a23c9ef ZFS allocates and frees pages frequently (via the ABD structure), e.g. for every disk access. The additional overhead of zeroing these pages is significant. For example, I measured a ~40% regression in performance of an uncached "zfs send ... >/dev/null". While this is a specific workload, this is a general problem because whenever we read from disk we are going to allocate pages, unnecessarily zero them, and then immediately fill them with the data from disk. I don't see any mechanism for ZFS to request that pages *not* be zeroed when they are allocated. This performance regression has been noted by users of ZFS on Ubuntu: https://github.com/zfsonlinux/zfs/issues/9910 I'd like to request that the default in Ubuntu be changed to init_on_alloc=0, so that users of ZFS on Ubuntu do not experience this performance regression. (FYI, my employer, Delphix, has worked around this by changing the default in our product, but my concern is that the vast majority of Ubuntu users will simply experience this as "Ubuntu HWE is much slower than before".) $ cat /proc/version_signature Ubuntu 5.3.0-28.30~18.04.1-generic 5.3.13 ** Affects: linux (Ubuntu) Importance: Undecided Status: New ** Attachment added: "lspci-vnvn.log" https://bugs.launchpad.net/bugs/1862822/+attachment/5327397/+files/lspci-vnvn.log -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1862822 Title: init_on_alloc=1 causes big performance regression for ZFS To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1862822/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1857398] Re: ubiquity should support encryption by default with zfsroot, with users able to opt in to running change-key after install
I agree with what Richard said above in comment #14, especially: > The installer should prompt (with a checkbox) for whether the user wants encryption. It should default to off. If the user selects the checkbox, prompt them for a passphrase. Setup encryption using that passphrase. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1857398 Title: ubiquity should support encryption by default with zfsroot, with users able to opt in to running change-key after install To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/1857398/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
