[Bug 1828248] Re: [linux-azure] Storage performance drop on RAID
This bug was fixed in the package linux-azure - 4.15.0-1045.49~14.04.1 --- linux-azure (4.15.0-1045.49~14.04.1) trusty; urgency=medium [ Ubuntu: 4.15.0-1045.49 ] * [linux-azure] Storage performance drop on RAID (LP: #1828248) - Revert "blk-mq: remove the request_list usage" * CVE-2018-12126 // CVE-2018-12127 // CVE-2018-12130 - Documentation/l1tf: Fix small spelling typo - x86/cpu: Sanitize FAM6_ATOM naming - kvm: x86: Report STIBP on GET_SUPPORTED_CPUID - locking/atomics, asm-generic: Move some macros from to a new file - tools include: Adopt linux/bits.h - x86/msr-index: Cleanup bit defines - x86/speculation: Consolidate CPU whitelists - x86/speculation/mds: Add basic bug infrastructure for MDS - x86/speculation/mds: Add BUG_MSBDS_ONLY - x86/kvm: Expose X86_FEATURE_MD_CLEAR to guests - x86/speculation/mds: Add mds_clear_cpu_buffers() - x86/speculation/mds: Clear CPU buffers on exit to user - x86/kvm/vmx: Add MDS protection when L1D Flush is not active - x86/speculation/mds: Conditionally clear CPU buffers on idle entry - x86/speculation/mds: Add mitigation control for MDS - x86/speculation/mds: Add sysfs reporting for MDS - x86/speculation/mds: Add mitigation mode VMWERV - Documentation: Move L1TF to separate directory - Documentation: Add MDS vulnerability documentation - x86/speculation/mds: Add mds=full,nosmt cmdline option - x86/speculation: Move arch_smt_update() call to after mitigation decisions - x86/speculation/mds: Add SMT warning message - x86/speculation/mds: Fix comment - x86/speculation/mds: Print SMT vulnerable on MSBDS with mitigations off - x86/speculation/mds: Add 'mitigations=' support for MDS * CVE-2017-5715 // CVE-2017-5753 - s390/speculation: Support 'mitigations=' cmdline option * CVE-2017-5715 // CVE-2017-5753 // CVE-2017-5754 // CVE-2018-3639 - powerpc/speculation: Support 'mitigations=' cmdline option * CVE-2017-5715 // CVE-2017-5754 // CVE-2018-3620 // CVE-2018-3639 // CVE-2018-3646 - cpu/speculation: Add 'mitigations=' cmdline option - x86/speculation: Support 'mitigations=' cmdline option * Packaging resync (LP: #1786013) - [Packaging] resync git-ubuntu-log linux-azure (4.15.0-1044.48~14.04.1) trusty; urgency=medium * linux-azure: 4.15.0-1044.48~14.04.1 -proposed tracker (LP: #1826352) * Packaging resync (LP: #1786013) - [Packaging] resync git-ubuntu-log - [Packaging] update helper scripts [ Ubuntu: 4.15.0-1044.48 ] * linux-azure: 4.15.0-1044.48 -proposed tracker (LP: #1826354) * [linux-azure] Include mainline commits fc96df16a1ce and ba50bf1ce9a5 in Azure kernel (LP: #1821378) - Drivers: hv: vmbus: Return -EINVAL for the sys files for unopened channels - Drivers: hv: vmbus: Check for ring when getting debug info * [linux-azure] Commit To Improve NVMe Performance (LP: #1819689) - blk-mq: remove the request_list usage * linux: 4.15.0-49.53 -proposed tracker (LP: #1826358) * Backport support for software count cache flush Spectre v2 mitigation. (CVE) (required for POWER9 DD2.3) (LP: #1822870) - powerpc/64s: Add support for ori barrier_nospec patching - powerpc/64s: Patch barrier_nospec in modules - powerpc/64s: Enable barrier_nospec based on firmware settings - powerpc: Use barrier_nospec in copy_from_user() - powerpc/64: Use barrier_nospec in syscall entry - powerpc/64s: Enhance the information in cpu_show_spectre_v1() - powerpc/64: Disable the speculation barrier from the command line - powerpc/64: Make stf barrier PPC_BOOK3S_64 specific. - powerpc/64: Add CONFIG_PPC_BARRIER_NOSPEC - powerpc/64: Call setup_barrier_nospec() from setup_arch() - powerpc/64: Make meltdown reporting Book3S 64 specific - powerpc/lib/code-patching: refactor patch_instruction() - powerpc/lib/feature-fixups: use raw_patch_instruction() - powerpc/asm: Add a patch_site macro & helpers for patching instructions - powerpc/64s: Add new security feature flags for count cache flush - powerpc/64s: Add support for software count cache flush - powerpc/pseries: Query hypervisor for count cache flush settings - powerpc/powernv: Query firmware for count cache flush settings - powerpc/fsl: Add nospectre_v2 command line argument - KVM: PPC: Book3S: Add count cache flush parameters to kvmppc_get_cpu_char() - [Config] Add CONFIG_PPC_BARRIER_NOSPEC * Packaging resync (LP: #1786013) - [Packaging] resync git-ubuntu-log * autopkgtests run too often, too much and don't skip enough (LP: #1823056) - [Debian] Set +x on rebuild testcase. - [Debian] Skip rebuild test, for regression-suite deps. - [Debian] Make ubuntu-regression-suite skippable on unbootable kernels. - [Debian] make rebuild use skippable error codes when skipping. - [Debian] Only run regression-suite, if requested to. * bionic: fork out
[Bug 1828248] Re: [linux-azure] Storage performance drop on RAID
This bug was fixed in the package linux-azure - 4.15.0-1045.49 --- linux-azure (4.15.0-1045.49) xenial; urgency=medium * [linux-azure] Storage performance drop on RAID (LP: #1828248) - Revert "blk-mq: remove the request_list usage" [ Ubuntu: 4.15.0-50.54 ] * CVE-2018-12126 // CVE-2018-12127 // CVE-2018-12130 - Documentation/l1tf: Fix small spelling typo - x86/cpu: Sanitize FAM6_ATOM naming - kvm: x86: Report STIBP on GET_SUPPORTED_CPUID - locking/atomics, asm-generic: Move some macros from to a new file - tools include: Adopt linux/bits.h - x86/msr-index: Cleanup bit defines - x86/speculation: Consolidate CPU whitelists - x86/speculation/mds: Add basic bug infrastructure for MDS - x86/speculation/mds: Add BUG_MSBDS_ONLY - x86/kvm: Expose X86_FEATURE_MD_CLEAR to guests - x86/speculation/mds: Add mds_clear_cpu_buffers() - x86/speculation/mds: Clear CPU buffers on exit to user - x86/kvm/vmx: Add MDS protection when L1D Flush is not active - x86/speculation/mds: Conditionally clear CPU buffers on idle entry - x86/speculation/mds: Add mitigation control for MDS - x86/speculation/mds: Add sysfs reporting for MDS - x86/speculation/mds: Add mitigation mode VMWERV - Documentation: Move L1TF to separate directory - Documentation: Add MDS vulnerability documentation - x86/speculation/mds: Add mds=full,nosmt cmdline option - x86/speculation: Move arch_smt_update() call to after mitigation decisions - x86/speculation/mds: Add SMT warning message - x86/speculation/mds: Fix comment - x86/speculation/mds: Print SMT vulnerable on MSBDS with mitigations off - x86/speculation/mds: Add 'mitigations=' support for MDS * CVE-2017-5715 // CVE-2017-5753 - s390/speculation: Support 'mitigations=' cmdline option * CVE-2017-5715 // CVE-2017-5753 // CVE-2017-5754 // CVE-2018-3639 - powerpc/speculation: Support 'mitigations=' cmdline option * CVE-2017-5715 // CVE-2017-5754 // CVE-2018-3620 // CVE-2018-3639 // CVE-2018-3646 - cpu/speculation: Add 'mitigations=' cmdline option - x86/speculation: Support 'mitigations=' cmdline option * Packaging resync (LP: #1786013) - [Packaging] resync git-ubuntu-log linux-azure (4.15.0-1044.48) xenial; urgency=medium * linux-azure: 4.15.0-1044.48 -proposed tracker (LP: #1826354) * [linux-azure] Include mainline commits fc96df16a1ce and ba50bf1ce9a5 in Azure kernel (LP: #1821378) - Drivers: hv: vmbus: Return -EINVAL for the sys files for unopened channels - Drivers: hv: vmbus: Check for ring when getting debug info * [linux-azure] Commit To Improve NVMe Performance (LP: #1819689) - blk-mq: remove the request_list usage [ Ubuntu: 4.15.0-49.53 ] * linux: 4.15.0-49.53 -proposed tracker (LP: #1826358) * Backport support for software count cache flush Spectre v2 mitigation. (CVE) (required for POWER9 DD2.3) (LP: #1822870) - powerpc/64s: Add support for ori barrier_nospec patching - powerpc/64s: Patch barrier_nospec in modules - powerpc/64s: Enable barrier_nospec based on firmware settings - powerpc: Use barrier_nospec in copy_from_user() - powerpc/64: Use barrier_nospec in syscall entry - powerpc/64s: Enhance the information in cpu_show_spectre_v1() - powerpc/64: Disable the speculation barrier from the command line - powerpc/64: Make stf barrier PPC_BOOK3S_64 specific. - powerpc/64: Add CONFIG_PPC_BARRIER_NOSPEC - powerpc/64: Call setup_barrier_nospec() from setup_arch() - powerpc/64: Make meltdown reporting Book3S 64 specific - powerpc/lib/code-patching: refactor patch_instruction() - powerpc/lib/feature-fixups: use raw_patch_instruction() - powerpc/asm: Add a patch_site macro & helpers for patching instructions - powerpc/64s: Add new security feature flags for count cache flush - powerpc/64s: Add support for software count cache flush - powerpc/pseries: Query hypervisor for count cache flush settings - powerpc/powernv: Query firmware for count cache flush settings - powerpc/fsl: Add nospectre_v2 command line argument - KVM: PPC: Book3S: Add count cache flush parameters to kvmppc_get_cpu_char() - [Config] Add CONFIG_PPC_BARRIER_NOSPEC * Packaging resync (LP: #1786013) - [Packaging] resync git-ubuntu-log * autopkgtests run too often, too much and don't skip enough (LP: #1823056) - [Debian] Set +x on rebuild testcase. - [Debian] Skip rebuild test, for regression-suite deps. - [Debian] Make ubuntu-regression-suite skippable on unbootable kernels. - [Debian] make rebuild use skippable error codes when skipping. - [Debian] Only run regression-suite, if requested to. * bionic: fork out linux-snapdragon into its own topic kernel (LP: #1820868) - [Packaging] remove arm64 snapdragon from getabis - [Config] config changes for snapdragon split - packaging: arm64: disable building the
[Bug 1828248] Re: [linux-azure] Storage performance drop on RAID
Yes, that is correct. Comment #6. Sorry for the typo. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828248 Title: [linux-azure] Storage performance drop on RAID To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1828248/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1828248] Re: [linux-azure] Storage performance drop on RAID
Morning Joe, On 5/14/19 8:40 AM, Joseph Salisbury wrote: > @marcelo, we do want the changes requested in comment #5 in bug 1819689. > We can wait for the results of Long Li's investigation before proceeding > there. > that is the revert in comment #6, right? Terry -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828248 Title: [linux-azure] Storage performance drop on RAID To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1828248/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828248] Re: [linux-azure] Storage performance drop on RAID
Thanks for the feedback, @jsalisbury. So we are proceeding with the revert for now. Later when Long Li's investigation is ready, can you open a new bug listing the remaining commits that need to be included? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828248 Title: [linux-azure] Storage performance drop on RAID To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1828248/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828248] Re: [linux-azure] Storage performance drop on RAID
@marcelo, we do want the changes requested in comment #5 in bug 1819689. We can wait for the results of Long Li's investigation before proceeding there. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828248 Title: [linux-azure] Storage performance drop on RAID To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1828248/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828248] Re: [linux-azure] Storage performance drop on RAID
@longli I would like your assessment of the changes that we've made here to get back to expected performance. I know we are not getting the same results on NVMe devices with 16.04 and 18.04 and I'm wondering if there's something substantially different with the 4.15 kernel in use on 16.04 that prevents us from making the same changes that we did with 4.18 to get better performance. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828248 Title: [linux-azure] Storage performance drop on RAID To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1828248/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828248] Re: [linux-azure] Storage performance drop on RAID
This patch was intended to increase performance, not reduce it. Is it too late to revert it? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828248 Title: [linux-azure] Storage performance drop on RAID To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1828248/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828248] Re: [linux-azure] Storage performance drop on RAID
** Tags added: patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828248 Title: [linux-azure] Storage performance drop on RAID To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1828248/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828248] Re: [linux-azure] Storage performance drop on RAID
Hi, Joe. Do you think the revert is enough for now, or should we discuss about eventyually including the changes that you had mentioned on the comment #5 in the bug #1819689? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828248 Title: [linux-azure] Storage performance drop on RAID To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1828248/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828248] Re: [linux-azure] Storage performance drop on RAID
Exactly, Joseph. I'm attaching the exact patch I used for building the test kernel. ** Patch added: "0001-Revert-blk-mq-remove-the-request_list-usage.patch" https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1828248/+attachment/5263471/+files/0001-Revert-blk-mq-remove-the-request_list-usage.patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828248 Title: [linux-azure] Storage performance drop on RAID To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1828248/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828248] Re: [linux-azure] Storage performance drop on RAID
@Marcelo Cerri, just to confirm, the test kernel posted in comment #6 only had commit 7ac257b862f2c reverted? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828248 Title: [linux-azure] Storage performance drop on RAID To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1828248/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828248] Re: [linux-azure] Storage performance drop on RAID
** Also affects: linux-azure (Ubuntu Xenial) Importance: Undecided Status: New ** Changed in: linux-azure (Ubuntu Xenial) Status: New => Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828248 Title: [linux-azure] Storage performance drop on RAID To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1828248/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828248] Re: [linux-azure] Storage performance drop on RAID
The test kernel is reaching 52K IOPS as expected. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828248 Title: [linux-azure] Storage performance drop on RAID To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1828248/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828248] Re: [linux-azure] Storage performance drop on RAID
I built a test kernel reverting the commit that Joe has mention: https://kernel.ubuntu.com/~mhcerri/azure/xenial-linux- azure-4.15.0-1044.48+1828248/ -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828248 Title: [linux-azure] Storage performance drop on RAID To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1828248/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828248] Re: [linux-azure] Storage performance drop on RAID
VM size is Standard_DS14_v2 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828248 Title: [linux-azure] Storage performance drop on RAID To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1828248/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828248] Re: [linux-azure] Storage performance drop on RAID
Adrian, can you add the VM size here, they are trying to reproduce. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828248 Title: [linux-azure] Storage performance drop on RAID To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1828248/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828248] Re: [linux-azure] Storage performance drop on RAID
This commit was also added to the Ubuntu-azure-4.18.0-1017 kernel. Are we seeing expected performing with that kernel? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828248 Title: [linux-azure] Storage performance drop on RAID To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1828248/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828248] Re: [linux-azure] Storage performance drop on RAID
The following commit was added in bug fa55b5d226dd. 7ac257b862f2c (“blk-mq: remove the request_list usage”) This commit cleans up unused code in block-mq (mq uses pre-allocated tags to allocate request, not request_list as it's used only in legacy queue). A test kernel with a revert of this commit would prove if it is the cause of this regression. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828248 Title: [linux-azure] Storage performance drop on RAID To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1828248/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828248] Re: [linux-azure] Storage performance drop on RAID
The bug number from comment one should have been bug 1819689 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828248 Title: [linux-azure] Storage performance drop on RAID To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1828248/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
