[Kernel-packages] [Bug 1319003] Re: Storage performance regression when Xen backend lacks persistent-grants support
** Description changed: Description of problem: When used as a Xen guest, Ubuntu 13.10 may be slower than older releases in terms of storage performance. This is due to the persistent-grants feature introduced in xen-blkfront on the Linux Kernel 3.8 series. From 3.8 to 3.12 (inclusive), xen-blkfront will add an extra set of memcpy() operations regardless of persistent-grants support in the backend (i.e. xen-blkback, qemu, tapdisk). Many Xen dom0s do not have backend persistent-grants support (such as Citrix XenServer and any Linux distro with Kernel prior to 3.8). This has been identified and fixed in the 3.13 kernel series [1], but was not backported to previous LTS kernels due to the nature of the bug (performance only). While persistent grants reduce the stress on the Xen grant table and allow for much better aggregate throughput (at the cost of an extra set of memcpy() operations), adding the copy overhead when the feature is unsupported on the backend combines the worst of both worlds. This is particularly noticeable when intensive storage workloads are active from many guests. The graphs attached show storage throughput numbers for Linux guests using kernel 3.12.9 (Graph 1) and 3.13.7 (Graph 2) running on a Citrix XenServer development build. The server had 4 storage repositories (SRs) with 1 Micron P320 SSD per SR (i.e. 10 VMs per SR means 40 VMs in total). When using 3.12.9 kernel, the regression is clearly visible for more than 2 VMs per SR and block sizes larger than 64 KiB. The workload consisted of sequential reads on pre-allocated raw LVM logical volumes. [1] Commits by Roger Pau Monné: bfe11d6de1c416cea4f3f0f35f864162063ce3fa fbe363c476afe8ec992d3baf682670a4bd1b6ce6 - Version-Release number of selected component (if applicable): xen-blkfront of Linux kernel 3.11 - How reproducible: This is always reproducible when a Ubuntu 13.10 guest is running on Xen and the storage backend (i.e. xen-blkback, qemu, tapdisk) does not have support for persistent grants. Steps to Reproduce: 1. Install a Xen dom0 running a kernel prior to 3.8 (without persistent-grants support). 2. Install a set of Ubuntu 13.10 guests (which uses kernel 3.11). 3. Measure aggregate storage throughput from all guests. NOTE: The storage infrastructure (e.g. local SSDs, network-attached storage) should not be a bottleneck in itself. If tested on a single SATA disk, for example, the issue will probably be unnoticeable as the infrastructure will be limiting response time and throughput. - Actual results: Aggregate storage throughput will be lower than with a xen-blkfront versions prior to 3.8 or newer than 3.12. - Expected results: - Aggregate storage throughput should be at least as good or better than previous versions of RHEL in cases where the backend doesn't support persistent grants. - + Aggregate storage throughput should be at least as good or better than previous (or newer) versions of Ubuntu in cases where the backend doesn't support persistent grants. Additional info: - Given that this is fixed on newer kernels, we urge that Red Hat request a backport of the relevant patches to the 3.11 stable branch. According to the rules in: https://www.kernel.org/doc/Documentation/stable_kernel_rules.txt, the patches would be accepted on the grounds of: + Given that this is fixed on newer kernels, we urge that a backport of the relevant patches to the 3.11 stable branch is requested. According to the rules in: https://www.kernel.org/doc/Documentation/stable_kernel_rules.txt, the patches would be accepted on the grounds of: - Serious issues as reported by a user of a distribution kernel may also be considered if they fix a notable performance or interactivity issue. As these fixes are not as obvious and have a higher risk of a subtle regression they should only be submitted by a distribution kernel maintainer and include an addendum linking to a bugzilla entry if it exists and additional information on the user-visible impact. ** Package changed: ubuntu = linux-meta (Ubuntu) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-meta in Ubuntu. https://bugs.launchpad.net/bugs/1319003 Title: Storage performance regression when Xen backend lacks persistent- grants support Status in “linux-meta” package in Ubuntu: New Bug description: Description of problem: When used as a Xen guest, Ubuntu 13.10 may be slower than older releases in terms of storage performance. This is due to the persistent-grants feature introduced in xen-blkfront on the Linux Kernel 3.8 series. From 3.8 to 3.12 (inclusive), xen-blkfront will add an extra set of memcpy() operations regardless of persistent-grants support in the backend (i.e. xen-blkback, qemu, tapdisk). Many Xen dom0s do not have backend persistent-grants support
[Kernel-packages] [Bug 1319003] Re: Storage performance regression when Xen backend lacks persistent-grants support
I am unable to collect logs with apport-collect as I do not have a set of Ubuntu guests setup for repro at the moment. I also believe logs are not relevant for this particular case. ** Changed in: linux (Ubuntu) Status: Incomplete = Confirmed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1319003 Title: Storage performance regression when Xen backend lacks persistent- grants support Status in “linux” package in Ubuntu: Confirmed Bug description: Description of problem: When used as a Xen guest, Ubuntu 13.10 may be slower than older releases in terms of storage performance. This is due to the persistent-grants feature introduced in xen-blkfront on the Linux Kernel 3.8 series. From 3.8 to 3.12 (inclusive), xen-blkfront will add an extra set of memcpy() operations regardless of persistent-grants support in the backend (i.e. xen-blkback, qemu, tapdisk). Many Xen dom0s do not have backend persistent-grants support (such as Citrix XenServer and any Linux distro with Kernel prior to 3.8). This has been identified and fixed in the 3.13 kernel series [1], but was not backported to previous LTS kernels due to the nature of the bug (performance only). While persistent grants reduce the stress on the Xen grant table and allow for much better aggregate throughput (at the cost of an extra set of memcpy() operations), adding the copy overhead when the feature is unsupported on the backend combines the worst of both worlds. This is particularly noticeable when intensive storage workloads are active from many guests. The graphs attached show storage throughput numbers for Linux guests using kernel 3.12.9 (Graph 1) and 3.13.7 (Graph 2) running on a Citrix XenServer development build. The server had 4 storage repositories (SRs) with 1 Micron P320 SSD per SR (i.e. 10 VMs per SR means 40 VMs in total). When using 3.12.9 kernel, the regression is clearly visible for more than 2 VMs per SR and block sizes larger than 64 KiB. The workload consisted of sequential reads on pre-allocated raw LVM logical volumes. [1] Commits by Roger Pau Monné: bfe11d6de1c416cea4f3f0f35f864162063ce3fa fbe363c476afe8ec992d3baf682670a4bd1b6ce6 Version-Release number of selected component (if applicable): xen-blkfront of Linux kernel 3.11 How reproducible: This is always reproducible when a Ubuntu 13.10 guest is running on Xen and the storage backend (i.e. xen-blkback, qemu, tapdisk) does not have support for persistent grants. Steps to Reproduce: 1. Install a Xen dom0 running a kernel prior to 3.8 (without persistent-grants support). 2. Install a set of Ubuntu 13.10 guests (which uses kernel 3.11). 3. Measure aggregate storage throughput from all guests. NOTE: The storage infrastructure (e.g. local SSDs, network-attached storage) should not be a bottleneck in itself. If tested on a single SATA disk, for example, the issue will probably be unnoticeable as the infrastructure will be limiting response time and throughput. Actual results: Aggregate storage throughput will be lower than with a xen-blkfront versions prior to 3.8 or newer than 3.12. Expected results: Aggregate storage throughput should be at least as good or better than previous (or newer) versions of Ubuntu in cases where the backend doesn't support persistent grants. Additional info: Given that this is fixed on newer kernels, we urge that a backport of the relevant patches to the 3.11 stable branch is requested. According to the rules in: https://www.kernel.org/doc/Documentation/stable_kernel_rules.txt, the patches would be accepted on the grounds of: - Serious issues as reported by a user of a distribution kernel may also be considered if they fix a notable performance or interactivity issue. As these fixes are not as obvious and have a higher risk of a subtle regression they should only be submitted by a distribution kernel maintainer and include an addendum linking to a bugzilla entry if it exists and additional information on the user-visible impact. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1319003/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1319003] Re: Storage performance regression when Xen backend lacks persistent-grants support
Joseph, apologies for the delay on this. I have finally managed to retest the kernel you have provided. For reference, I added the following disks to my server: * 4 x Micron P320 * 2 x Intel 910s (presented as 4 separate SCSI devices) * 1 x Fusion-io ioDrive2 I created 10 LVM Logical Volumes on each one of these 9 devices. Next, I created 10 Saucy 64bit VMs (each with 2 vCPUs and 512 MB RAM) and assigned one LV from each device to it. Each data point in the graphs I am attaching now correspond to the aggregate throughput of all LVs per VM. Your kernel (indicated by Ubuntu 13.10 x86_64 + Backports) allows the VMs to reach the 7 GB/s mark while the kernel without the backports will not go past the 5 GB/s mark. I hope this confirms that these patches are essential and that you can include them as soon as possible. Regards, Felipe -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1319003 Title: Storage performance regression when Xen backend lacks persistent- grants support Status in “linux” package in Ubuntu: In Progress Status in “linux” source package in Saucy: In Progress Bug description: Description of problem: When used as a Xen guest, Ubuntu 13.10 may be slower than older releases in terms of storage performance. This is due to the persistent-grants feature introduced in xen-blkfront on the Linux Kernel 3.8 series. From 3.8 to 3.12 (inclusive), xen-blkfront will add an extra set of memcpy() operations regardless of persistent-grants support in the backend (i.e. xen-blkback, qemu, tapdisk). Many Xen dom0s do not have backend persistent-grants support (such as Citrix XenServer and any Linux distro with Kernel prior to 3.8). This has been identified and fixed in the 3.13 kernel series [1], but was not backported to previous LTS kernels due to the nature of the bug (performance only). While persistent grants reduce the stress on the Xen grant table and allow for much better aggregate throughput (at the cost of an extra set of memcpy() operations), adding the copy overhead when the feature is unsupported on the backend combines the worst of both worlds. This is particularly noticeable when intensive storage workloads are active from many guests. The graphs attached show storage throughput numbers for Linux guests using kernel 3.12.9 (Graph 1) and 3.13.7 (Graph 2) running on a Citrix XenServer development build. The server had 4 storage repositories (SRs) with 1 Micron P320 SSD per SR (i.e. 10 VMs per SR means 40 VMs in total). When using 3.12.9 kernel, the regression is clearly visible for more than 2 VMs per SR and block sizes larger than 64 KiB. The workload consisted of sequential reads on pre-allocated raw LVM logical volumes. [1] Commits by Roger Pau Monné: bfe11d6de1c416cea4f3f0f35f864162063ce3fa fbe363c476afe8ec992d3baf682670a4bd1b6ce6 Version-Release number of selected component (if applicable): xen-blkfront of Linux kernel 3.11 How reproducible: This is always reproducible when a Ubuntu 13.10 guest is running on Xen and the storage backend (i.e. xen-blkback, qemu, tapdisk) does not have support for persistent grants. Steps to Reproduce: 1. Install a Xen dom0 running a kernel prior to 3.8 (without persistent-grants support). 2. Install a set of Ubuntu 13.10 guests (which uses kernel 3.11). 3. Measure aggregate storage throughput from all guests. NOTE: The storage infrastructure (e.g. local SSDs, network-attached storage) should not be a bottleneck in itself. If tested on a single SATA disk, for example, the issue will probably be unnoticeable as the infrastructure will be limiting response time and throughput. Actual results: Aggregate storage throughput will be lower than with a xen-blkfront versions prior to 3.8 or newer than 3.12. Expected results: Aggregate storage throughput should be at least as good or better than previous (or newer) versions of Ubuntu in cases where the backend doesn't support persistent grants. Additional info: Given that this is fixed on newer kernels, we urge that a backport of the relevant patches to the 3.11 stable branch is requested. According to the rules in: https://www.kernel.org/doc/Documentation/stable_kernel_rules.txt, the patches would be accepted on the grounds of: - Serious issues as reported by a user of a distribution kernel may also be considered if they fix a notable performance or interactivity issue. As these fixes are not as obvious and have a higher risk of a subtle regression they should only be submitted by a distribution kernel maintainer and include an addendum linking to a bugzilla entry if it exists and additional information on the user-visible impact. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1319003/+subscriptions -- Mailing
[Kernel-packages] [Bug 1319003] Re: Storage performance regression when Xen backend lacks persistent-grants support
** Attachment added: Saucy x86_64 without the backports https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1319003/+attachment/4123531/+files/saucy.png -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1319003 Title: Storage performance regression when Xen backend lacks persistent- grants support Status in “linux” package in Ubuntu: In Progress Status in “linux” source package in Saucy: In Progress Bug description: Description of problem: When used as a Xen guest, Ubuntu 13.10 may be slower than older releases in terms of storage performance. This is due to the persistent-grants feature introduced in xen-blkfront on the Linux Kernel 3.8 series. From 3.8 to 3.12 (inclusive), xen-blkfront will add an extra set of memcpy() operations regardless of persistent-grants support in the backend (i.e. xen-blkback, qemu, tapdisk). Many Xen dom0s do not have backend persistent-grants support (such as Citrix XenServer and any Linux distro with Kernel prior to 3.8). This has been identified and fixed in the 3.13 kernel series [1], but was not backported to previous LTS kernels due to the nature of the bug (performance only). While persistent grants reduce the stress on the Xen grant table and allow for much better aggregate throughput (at the cost of an extra set of memcpy() operations), adding the copy overhead when the feature is unsupported on the backend combines the worst of both worlds. This is particularly noticeable when intensive storage workloads are active from many guests. The graphs attached show storage throughput numbers for Linux guests using kernel 3.12.9 (Graph 1) and 3.13.7 (Graph 2) running on a Citrix XenServer development build. The server had 4 storage repositories (SRs) with 1 Micron P320 SSD per SR (i.e. 10 VMs per SR means 40 VMs in total). When using 3.12.9 kernel, the regression is clearly visible for more than 2 VMs per SR and block sizes larger than 64 KiB. The workload consisted of sequential reads on pre-allocated raw LVM logical volumes. [1] Commits by Roger Pau Monné: bfe11d6de1c416cea4f3f0f35f864162063ce3fa fbe363c476afe8ec992d3baf682670a4bd1b6ce6 Version-Release number of selected component (if applicable): xen-blkfront of Linux kernel 3.11 How reproducible: This is always reproducible when a Ubuntu 13.10 guest is running on Xen and the storage backend (i.e. xen-blkback, qemu, tapdisk) does not have support for persistent grants. Steps to Reproduce: 1. Install a Xen dom0 running a kernel prior to 3.8 (without persistent-grants support). 2. Install a set of Ubuntu 13.10 guests (which uses kernel 3.11). 3. Measure aggregate storage throughput from all guests. NOTE: The storage infrastructure (e.g. local SSDs, network-attached storage) should not be a bottleneck in itself. If tested on a single SATA disk, for example, the issue will probably be unnoticeable as the infrastructure will be limiting response time and throughput. Actual results: Aggregate storage throughput will be lower than with a xen-blkfront versions prior to 3.8 or newer than 3.12. Expected results: Aggregate storage throughput should be at least as good or better than previous (or newer) versions of Ubuntu in cases where the backend doesn't support persistent grants. Additional info: Given that this is fixed on newer kernels, we urge that a backport of the relevant patches to the 3.11 stable branch is requested. According to the rules in: https://www.kernel.org/doc/Documentation/stable_kernel_rules.txt, the patches would be accepted on the grounds of: - Serious issues as reported by a user of a distribution kernel may also be considered if they fix a notable performance or interactivity issue. As these fixes are not as obvious and have a higher risk of a subtle regression they should only be submitted by a distribution kernel maintainer and include an addendum linking to a bugzilla entry if it exists and additional information on the user-visible impact. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1319003/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1319003] Re: Storage performance regression when Xen backend lacks persistent-grants support
** Attachment added: Saucy x86_64 with the backports https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1319003/+attachment/4123532/+files/saucy-backports.png -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1319003 Title: Storage performance regression when Xen backend lacks persistent- grants support Status in “linux” package in Ubuntu: In Progress Status in “linux” source package in Saucy: In Progress Bug description: Description of problem: When used as a Xen guest, Ubuntu 13.10 may be slower than older releases in terms of storage performance. This is due to the persistent-grants feature introduced in xen-blkfront on the Linux Kernel 3.8 series. From 3.8 to 3.12 (inclusive), xen-blkfront will add an extra set of memcpy() operations regardless of persistent-grants support in the backend (i.e. xen-blkback, qemu, tapdisk). Many Xen dom0s do not have backend persistent-grants support (such as Citrix XenServer and any Linux distro with Kernel prior to 3.8). This has been identified and fixed in the 3.13 kernel series [1], but was not backported to previous LTS kernels due to the nature of the bug (performance only). While persistent grants reduce the stress on the Xen grant table and allow for much better aggregate throughput (at the cost of an extra set of memcpy() operations), adding the copy overhead when the feature is unsupported on the backend combines the worst of both worlds. This is particularly noticeable when intensive storage workloads are active from many guests. The graphs attached show storage throughput numbers for Linux guests using kernel 3.12.9 (Graph 1) and 3.13.7 (Graph 2) running on a Citrix XenServer development build. The server had 4 storage repositories (SRs) with 1 Micron P320 SSD per SR (i.e. 10 VMs per SR means 40 VMs in total). When using 3.12.9 kernel, the regression is clearly visible for more than 2 VMs per SR and block sizes larger than 64 KiB. The workload consisted of sequential reads on pre-allocated raw LVM logical volumes. [1] Commits by Roger Pau Monné: bfe11d6de1c416cea4f3f0f35f864162063ce3fa fbe363c476afe8ec992d3baf682670a4bd1b6ce6 Version-Release number of selected component (if applicable): xen-blkfront of Linux kernel 3.11 How reproducible: This is always reproducible when a Ubuntu 13.10 guest is running on Xen and the storage backend (i.e. xen-blkback, qemu, tapdisk) does not have support for persistent grants. Steps to Reproduce: 1. Install a Xen dom0 running a kernel prior to 3.8 (without persistent-grants support). 2. Install a set of Ubuntu 13.10 guests (which uses kernel 3.11). 3. Measure aggregate storage throughput from all guests. NOTE: The storage infrastructure (e.g. local SSDs, network-attached storage) should not be a bottleneck in itself. If tested on a single SATA disk, for example, the issue will probably be unnoticeable as the infrastructure will be limiting response time and throughput. Actual results: Aggregate storage throughput will be lower than with a xen-blkfront versions prior to 3.8 or newer than 3.12. Expected results: Aggregate storage throughput should be at least as good or better than previous (or newer) versions of Ubuntu in cases where the backend doesn't support persistent grants. Additional info: Given that this is fixed on newer kernels, we urge that a backport of the relevant patches to the 3.11 stable branch is requested. According to the rules in: https://www.kernel.org/doc/Documentation/stable_kernel_rules.txt, the patches would be accepted on the grounds of: - Serious issues as reported by a user of a distribution kernel may also be considered if they fix a notable performance or interactivity issue. As these fixes are not as obvious and have a higher risk of a subtle regression they should only be submitted by a distribution kernel maintainer and include an addendum linking to a bugzilla entry if it exists and additional information on the user-visible impact. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1319003/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1319003] Re: Storage performance regression when Xen backend lacks persistent-grants support
** Attachment removed: Saucy x86_64 without the backports https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1319003/+attachment/4123531/+files/saucy.png ** Attachment removed: Saucy x86_64 with the backports https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1319003/+attachment/4123532/+files/saucy-backports.png ** Attachment added: Saucy x86_64 guests with the backports https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1319003/+attachment/4123554/+files/saucy64-backports.png -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1319003 Title: Storage performance regression when Xen backend lacks persistent- grants support Status in “linux” package in Ubuntu: In Progress Status in “linux” source package in Saucy: In Progress Bug description: Description of problem: When used as a Xen guest, Ubuntu 13.10 may be slower than older releases in terms of storage performance. This is due to the persistent-grants feature introduced in xen-blkfront on the Linux Kernel 3.8 series. From 3.8 to 3.12 (inclusive), xen-blkfront will add an extra set of memcpy() operations regardless of persistent-grants support in the backend (i.e. xen-blkback, qemu, tapdisk). Many Xen dom0s do not have backend persistent-grants support (such as Citrix XenServer and any Linux distro with Kernel prior to 3.8). This has been identified and fixed in the 3.13 kernel series [1], but was not backported to previous LTS kernels due to the nature of the bug (performance only). While persistent grants reduce the stress on the Xen grant table and allow for much better aggregate throughput (at the cost of an extra set of memcpy() operations), adding the copy overhead when the feature is unsupported on the backend combines the worst of both worlds. This is particularly noticeable when intensive storage workloads are active from many guests. The graphs attached show storage throughput numbers for Linux guests using kernel 3.12.9 (Graph 1) and 3.13.7 (Graph 2) running on a Citrix XenServer development build. The server had 4 storage repositories (SRs) with 1 Micron P320 SSD per SR (i.e. 10 VMs per SR means 40 VMs in total). When using 3.12.9 kernel, the regression is clearly visible for more than 2 VMs per SR and block sizes larger than 64 KiB. The workload consisted of sequential reads on pre-allocated raw LVM logical volumes. [1] Commits by Roger Pau Monné: bfe11d6de1c416cea4f3f0f35f864162063ce3fa fbe363c476afe8ec992d3baf682670a4bd1b6ce6 Version-Release number of selected component (if applicable): xen-blkfront of Linux kernel 3.11 How reproducible: This is always reproducible when a Ubuntu 13.10 guest is running on Xen and the storage backend (i.e. xen-blkback, qemu, tapdisk) does not have support for persistent grants. Steps to Reproduce: 1. Install a Xen dom0 running a kernel prior to 3.8 (without persistent-grants support). 2. Install a set of Ubuntu 13.10 guests (which uses kernel 3.11). 3. Measure aggregate storage throughput from all guests. NOTE: The storage infrastructure (e.g. local SSDs, network-attached storage) should not be a bottleneck in itself. If tested on a single SATA disk, for example, the issue will probably be unnoticeable as the infrastructure will be limiting response time and throughput. Actual results: Aggregate storage throughput will be lower than with a xen-blkfront versions prior to 3.8 or newer than 3.12. Expected results: Aggregate storage throughput should be at least as good or better than previous (or newer) versions of Ubuntu in cases where the backend doesn't support persistent grants. Additional info: Given that this is fixed on newer kernels, we urge that a backport of the relevant patches to the 3.11 stable branch is requested. According to the rules in: https://www.kernel.org/doc/Documentation/stable_kernel_rules.txt, the patches would be accepted on the grounds of: - Serious issues as reported by a user of a distribution kernel may also be considered if they fix a notable performance or interactivity issue. As these fixes are not as obvious and have a higher risk of a subtle regression they should only be submitted by a distribution kernel maintainer and include an addendum linking to a bugzilla entry if it exists and additional information on the user-visible impact. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1319003/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1319003] Re: Storage performance regression when Xen backend lacks persistent-grants support
** Attachment added: Saucy x86_64 guests without the backports https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1319003/+attachment/4123555/+files/saucy64.png -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1319003 Title: Storage performance regression when Xen backend lacks persistent- grants support Status in “linux” package in Ubuntu: In Progress Status in “linux” source package in Saucy: In Progress Bug description: Description of problem: When used as a Xen guest, Ubuntu 13.10 may be slower than older releases in terms of storage performance. This is due to the persistent-grants feature introduced in xen-blkfront on the Linux Kernel 3.8 series. From 3.8 to 3.12 (inclusive), xen-blkfront will add an extra set of memcpy() operations regardless of persistent-grants support in the backend (i.e. xen-blkback, qemu, tapdisk). Many Xen dom0s do not have backend persistent-grants support (such as Citrix XenServer and any Linux distro with Kernel prior to 3.8). This has been identified and fixed in the 3.13 kernel series [1], but was not backported to previous LTS kernels due to the nature of the bug (performance only). While persistent grants reduce the stress on the Xen grant table and allow for much better aggregate throughput (at the cost of an extra set of memcpy() operations), adding the copy overhead when the feature is unsupported on the backend combines the worst of both worlds. This is particularly noticeable when intensive storage workloads are active from many guests. The graphs attached show storage throughput numbers for Linux guests using kernel 3.12.9 (Graph 1) and 3.13.7 (Graph 2) running on a Citrix XenServer development build. The server had 4 storage repositories (SRs) with 1 Micron P320 SSD per SR (i.e. 10 VMs per SR means 40 VMs in total). When using 3.12.9 kernel, the regression is clearly visible for more than 2 VMs per SR and block sizes larger than 64 KiB. The workload consisted of sequential reads on pre-allocated raw LVM logical volumes. [1] Commits by Roger Pau Monné: bfe11d6de1c416cea4f3f0f35f864162063ce3fa fbe363c476afe8ec992d3baf682670a4bd1b6ce6 Version-Release number of selected component (if applicable): xen-blkfront of Linux kernel 3.11 How reproducible: This is always reproducible when a Ubuntu 13.10 guest is running on Xen and the storage backend (i.e. xen-blkback, qemu, tapdisk) does not have support for persistent grants. Steps to Reproduce: 1. Install a Xen dom0 running a kernel prior to 3.8 (without persistent-grants support). 2. Install a set of Ubuntu 13.10 guests (which uses kernel 3.11). 3. Measure aggregate storage throughput from all guests. NOTE: The storage infrastructure (e.g. local SSDs, network-attached storage) should not be a bottleneck in itself. If tested on a single SATA disk, for example, the issue will probably be unnoticeable as the infrastructure will be limiting response time and throughput. Actual results: Aggregate storage throughput will be lower than with a xen-blkfront versions prior to 3.8 or newer than 3.12. Expected results: Aggregate storage throughput should be at least as good or better than previous (or newer) versions of Ubuntu in cases where the backend doesn't support persistent grants. Additional info: Given that this is fixed on newer kernels, we urge that a backport of the relevant patches to the 3.11 stable branch is requested. According to the rules in: https://www.kernel.org/doc/Documentation/stable_kernel_rules.txt, the patches would be accepted on the grounds of: - Serious issues as reported by a user of a distribution kernel may also be considered if they fix a notable performance or interactivity issue. As these fixes are not as obvious and have a higher risk of a subtle regression they should only be submitted by a distribution kernel maintainer and include an addendum linking to a bugzilla entry if it exists and additional information on the user-visible impact. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1319003/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1775235] Re: Ubuntu 16.04 (4.4.0-127) hangs on boot with virtio-scsi MQ enabled
** Description changed:
We noticed that Ubuntu 16.04 guests running on Nutanix AHV stopped
booting after they were upgraded to the latest kernel (4.4.0-127). Only
guests with scsi mq enabled suffered from this problem. AHV is one of
the few hypervisor products to offer multiqueue for virtio-scsi devices.
Upon further investigation, we could see that the kernel would hang
during the scanning of scsi targets. More specifically, immediately
after coming across a target without any luns present. That's the first
time the kernel destroys a target (given it doesn't have luns). This
could be confirmed with gdb (attached to qemu's gdbserver):
#0 0xc0045039 in ?? ()
#1 0x88022c753c98 in ?? ()
#2 0x815d1de6 in scsi_target_destroy (starget=0x88022ad62400)
- at /build/linux-E14mqW/linux-4.4.0/drivers/scsi/scsi_scan.c:322
+ at /build/linux-E14mqW/linux-4.4.0/drivers/scsi/scsi_scan.c:322
This shows the guest vCPU stuck on virtio-scsi's implementation of
target_destroy. Despite lacking symbols, we managed to examine the
virtio_scsi_target_state to see that the 'reqs' counter was invalid:
(gdb) p *(struct virtio_scsi_target_state *)starget->hostdata
$6 = {tgt_seq = {sequence = 0}, reqs = {counter = -1}, req_vq =
0x88022cbdd9e8}
(gdb)
This drew our attention to the following patch which is exclusive to the
Ubuntu kernel:
commit f1f609d8015e1d34d39458924dcd9524fccd4307
Author: Jay Vosburgh
Date: Thu Apr 19 21:40:00 2018 +0200
In a nutshell, the patch spins on the target's 'reqs' counter waiting for the
target to quiesce:
--- a/drivers/scsi/virtio_scsi.c
+++ b/drivers/scsi/virtio_scsi.c
@@ -785,6 +785,10 @@ static int virtscsi_target_alloc(struct scsi_target
*starget)
- static void virtscsi_target_destroy(struct scsi_target *starget)
- {
- struct virtio_scsi_target_state *tgt = starget->hostdata;
+ static void virtscsi_target_destroy(struct scsi_target *starget)
+ {
+ struct virtio_scsi_target_state *tgt = starget->hostdata;
+
+ /* we can race with concurrent virtscsi_complete_cmd */
+ while (atomic_read(>reqs))
+ cpu_relax();
- kfree(tgt);
- }
+ kfree(tgt);
+ }
Personally, I think this is a catastrophic way of waiting for a target
to quiesce since virtscsi_target_destroy() is called with IRQs disabled
from scsi_scan.c:scsi_target_destroy(). Devices which take a long time
to quiesce during a target_destroy() could hog the CPU for relatively
long periods of time.
Nevertheless, further study revealed that virtio-scsi itself is broken
in a way that it doesn't increment the 'reqs' counter when submitting
requests on MQ in certain conditions. That caused the counter to go to
-1 (on the completion of the first request) and the CPU to hang
indefinitely.
The following patch fixes the issue:
- --- new/linux-4.4.0/drivers/scsi/virtio_scsi.c2018-06-05
10:03:29.083428545 -0700
- +++ old/linux-4.4.0/drivers/scsi/virtio_scsi.c2018-06-04
10:23:07.0 -0700
- @@ -641,10 +641,9 @@
+ --- old/linux-4.4.0/drivers/scsi/virtio_scsi.c2018-06-04
10:23:07.0 -0700
+ +++ new/linux-4.4.0/drivers/scsi/virtio_scsi.c2018-06-05
10:03:29.083428545 -0700
+ @@ -641,9 +641,10 @@
scsi_target(sc->device)->hostdata;
struct virtio_scsi_vq *req_vq;
- - if (shost_use_blk_mq(sh)) {
- + if (shost_use_blk_mq(sh))
+ - if (shost_use_blk_mq(sh))
+ + if (shost_use_blk_mq(sh)) {
req_vq = virtscsi_pick_vq_mq(vscsi, sc);
- - atomic_inc(>reqs);
- - } else
- + else
+ - else
+ + atomic_inc(>reqs);
+ + } else
req_vq = virtscsi_pick_vq(vscsi, tgt);
return virtscsi_queuecommand(vscsi, req_vq, sc);
Signed-off-by: Felipe Franciosi
-
- Please consider this a urgent fix as all of our customers which use Ubuntu
16.04 and have MQ enabled for better performance will be affected by your
latest update. Our workaround is to recommend that they disable SCSI MQ while
you work on the issue.
+ Please consider this a urgent fix as all of our customers which use
+ Ubuntu 16.04 and have MQ enabled for better performance will be affected
+ by your latest update. Our workaround is to recommend that they disable
+ SCSI MQ while you work on the issue.
Best regards,
Felipe
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1775235
Title:
Ubuntu 16.04 (4.4.0-127) hangs on boot with virtio-scsi MQ enabled
Status in linux package in Ubuntu:
Confirmed
Bug description:
We noticed that Ubuntu 16.04 guests running on Nutanix AHV stopped
booting after they were upgraded to the latest kernel (4.4.0-127).
Only gue
[Kernel-packages] [Bug 1775235] Re: Ubuntu 16.04 (4.4.0-127) hangs on boot with virtio-scsi MQ enabled
Hi Joseph,
We gave it a try internally and it looks good. Passed the same set of
tests I ran with the previous kernel you provided: works with mq
disabled, enabled and also passes a simple io integrity test.
Thanks,
F.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1775235
Title:
Ubuntu 16.04 (4.4.0-127) hangs on boot with virtio-scsi MQ enabled
Status in linux package in Ubuntu:
Invalid
Status in linux source package in Xenial:
In Progress
Bug description:
We noticed that Ubuntu 16.04 guests running on Nutanix AHV stopped
booting after they were upgraded to the latest kernel (4.4.0-127).
Only guests with scsi mq enabled suffered from this problem. AHV is
one of the few hypervisor products to offer multiqueue for virtio-scsi
devices.
Upon further investigation, we could see that the kernel would hang
during the scanning of scsi targets. More specifically, immediately
after coming across a target without any luns present. That's the
first time the kernel destroys a target (given it doesn't have luns).
This could be confirmed with gdb (attached to qemu's gdbserver):
#0 0xc0045039 in ?? ()
#1 0x88022c753c98 in ?? ()
#2 0x815d1de6 in scsi_target_destroy (starget=0x88022ad62400)
at /build/linux-E14mqW/linux-4.4.0/drivers/scsi/scsi_scan.c:322
This shows the guest vCPU stuck on virtio-scsi's implementation of
target_destroy. Despite lacking symbols, we managed to examine the
virtio_scsi_target_state to see that the 'reqs' counter was invalid:
(gdb) p *(struct virtio_scsi_target_state *)starget->hostdata
$6 = {tgt_seq = {sequence = 0}, reqs = {counter = -1}, req_vq =
0x88022cbdd9e8}
(gdb)
This drew our attention to the following patch which is exclusive to the
Ubuntu kernel:
commit f1f609d8015e1d34d39458924dcd9524fccd4307
Author: Jay Vosburgh
Date: Thu Apr 19 21:40:00 2018 +0200
In a nutshell, the patch spins on the target's 'reqs' counter waiting for the
target to quiesce:
--- a/drivers/scsi/virtio_scsi.c
+++ b/drivers/scsi/virtio_scsi.c
@@ -785,6 +785,10 @@ static int virtscsi_target_alloc(struct scsi_target
*starget)
static void virtscsi_target_destroy(struct scsi_target *starget)
{
struct virtio_scsi_target_state *tgt = starget->hostdata;
+
+ /* we can race with concurrent virtscsi_complete_cmd */
+ while (atomic_read(>reqs))
+ cpu_relax();
kfree(tgt);
}
Personally, I think this is a catastrophic way of waiting for a target
to quiesce since virtscsi_target_destroy() is called with IRQs
disabled from scsi_scan.c:scsi_target_destroy(). Devices which take a
long time to quiesce during a target_destroy() could hog the CPU for
relatively long periods of time.
Nevertheless, further study revealed that virtio-scsi itself is broken
in a way that it doesn't increment the 'reqs' counter when submitting
requests on MQ in certain conditions. That caused the counter to go to
-1 (on the completion of the first request) and the CPU to hang
indefinitely.
The following patch fixes the issue:
--- old/linux-4.4.0/drivers/scsi/virtio_scsi.c2018-06-04
10:23:07.0 -0700
+++ new/linux-4.4.0/drivers/scsi/virtio_scsi.c2018-06-05
10:03:29.083428545 -0700
@@ -641,9 +641,10 @@
scsi_target(sc->device)->hostdata;
struct virtio_scsi_vq *req_vq;
- if (shost_use_blk_mq(sh))
+ if (shost_use_blk_mq(sh)) {
req_vq = virtscsi_pick_vq_mq(vscsi, sc);
- else
+ atomic_inc(>reqs);
+ } else
req_vq = virtscsi_pick_vq(vscsi, tgt);
return virtscsi_queuecommand(vscsi, req_vq, sc);
Signed-off-by: Felipe Franciosi
Please consider this a urgent fix as all of our customers which use
Ubuntu 16.04 and have MQ enabled for better performance will be
affected by your latest update. Our workaround is to recommend that
they disable SCSI MQ while you work on the issue.
Best regards,
Felipe
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1775235/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1775235] Re: Ubuntu 16.04 (4.4.0-127) hangs on boot with virtio-scsi MQ enabled
Hi Joseph,
Thanks for looking at this so promptly. I have downloaded and tested
your kernel. It appears to work well both with scsi mq enabled and
disabled (over virtio-scsi). I also ran some basic io integrity tests
and didn't spot any problems.
Before you commit this, may I propose a v2 of my own patch which is
functionally identical but slightly more elegant? It does the
atomic_inc() within virtscsi_pick_vq_mq(). That's probably preferable
given the other uses of atomic_*() for non-mq are done within
virtscsi_pick_vq().
Have a look:
-8<-
--- old/linux-4.4.0/drivers/scsi/virtio_scsi.c2018-06-04
10:23:07.0 -0700
+++ new/linux-4.4.0/drivers/scsi/virtio_scsi.c2018-06-07
06:20:58.596764040 -0700
@@ -588,11 +588,13 @@
}
static struct virtio_scsi_vq *virtscsi_pick_vq_mq(struct virtio_scsi *vscsi,
+ struct
virtio_scsi_target_state *tgt,
struct scsi_cmnd *sc)
{
u32 tag = blk_mq_unique_tag(sc->request);
u16 hwq = blk_mq_unique_tag_to_hwq(tag);
+ atomic_inc(>reqs);
return >req_vqs[hwq];
}
@@ -642,7 +644,7 @@
struct virtio_scsi_vq *req_vq;
if (shost_use_blk_mq(sh))
- req_vq = virtscsi_pick_vq_mq(vscsi, sc);
+ req_vq = virtscsi_pick_vq_mq(vscsi, tgt, sc);
else
req_vq = virtscsi_pick_vq(vscsi, tgt);
Signed-off-by: Felipe Franciosi
-8<-
I'm happy to test it again if you'd like, but it should be functionally
identical.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1775235
Title:
Ubuntu 16.04 (4.4.0-127) hangs on boot with virtio-scsi MQ enabled
Status in linux package in Ubuntu:
Invalid
Status in linux source package in Xenial:
In Progress
Bug description:
We noticed that Ubuntu 16.04 guests running on Nutanix AHV stopped
booting after they were upgraded to the latest kernel (4.4.0-127).
Only guests with scsi mq enabled suffered from this problem. AHV is
one of the few hypervisor products to offer multiqueue for virtio-scsi
devices.
Upon further investigation, we could see that the kernel would hang
during the scanning of scsi targets. More specifically, immediately
after coming across a target without any luns present. That's the
first time the kernel destroys a target (given it doesn't have luns).
This could be confirmed with gdb (attached to qemu's gdbserver):
#0 0xc0045039 in ?? ()
#1 0x88022c753c98 in ?? ()
#2 0x815d1de6 in scsi_target_destroy (starget=0x88022ad62400)
at /build/linux-E14mqW/linux-4.4.0/drivers/scsi/scsi_scan.c:322
This shows the guest vCPU stuck on virtio-scsi's implementation of
target_destroy. Despite lacking symbols, we managed to examine the
virtio_scsi_target_state to see that the 'reqs' counter was invalid:
(gdb) p *(struct virtio_scsi_target_state *)starget->hostdata
$6 = {tgt_seq = {sequence = 0}, reqs = {counter = -1}, req_vq =
0x88022cbdd9e8}
(gdb)
This drew our attention to the following patch which is exclusive to the
Ubuntu kernel:
commit f1f609d8015e1d34d39458924dcd9524fccd4307
Author: Jay Vosburgh
Date: Thu Apr 19 21:40:00 2018 +0200
In a nutshell, the patch spins on the target's 'reqs' counter waiting for the
target to quiesce:
--- a/drivers/scsi/virtio_scsi.c
+++ b/drivers/scsi/virtio_scsi.c
@@ -785,6 +785,10 @@ static int virtscsi_target_alloc(struct scsi_target
*starget)
static void virtscsi_target_destroy(struct scsi_target *starget)
{
struct virtio_scsi_target_state *tgt = starget->hostdata;
+
+ /* we can race with concurrent virtscsi_complete_cmd */
+ while (atomic_read(>reqs))
+ cpu_relax();
kfree(tgt);
}
Personally, I think this is a catastrophic way of waiting for a target
to quiesce since virtscsi_target_destroy() is called with IRQs
disabled from scsi_scan.c:scsi_target_destroy(). Devices which take a
long time to quiesce during a target_destroy() could hog the CPU for
relatively long periods of time.
Nevertheless, further study revealed that virtio-scsi itself is broken
in a way that it doesn't increment the 'reqs' counter when submitting
requests on MQ in certain conditions. That caused the counter to go to
-1 (on the completion of the first request) and the CPU to hang
indefinitely.
The following patch fixes the issue:
--- old/linux-4.4.0/drivers/scsi/virtio_scsi.c2018-06-04
10:23:07.0 -0700
+++ new/linux-4.4.0/drivers/scsi/virtio_scsi.c2018-06-05
10:03:29.083428545 -0700
@@ -641,9 +641,10 @@
scsi_target(sc->device)->hostdata;
struct virtio_scsi_vq *req_vq;
-
[Kernel-packages] [Bug 1775235] Re: Ubuntu 16.04 (4.4.0-127) hangs on boot with virtio-scsi MQ enabled
scsi_target(sc->device)->hostdata;
struct virtio_scsi_vq *req_vq;
- if (shost_use_blk_mq(sh))
+ if (shost_use_blk_mq(sh)) {
req_vq = virtscsi_pick_vq_mq(vscsi, sc);
- else
+ atomic_inc(>reqs);
+ } else
req_vq = virtscsi_pick_vq(vscsi, tgt);
return virtscsi_queuecommand(vscsi, req_vq, sc);
Signed-off-by: Felipe Franciosi
Please consider this a urgent fix as all of our customers which use
Ubuntu 16.04 and have MQ enabled for better performance will be
affected by your latest update. Our workaround is to recommend that
they disable SCSI MQ while you work on the issue.
Best regards,
Felipe
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1775235/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1775235] [NEW] Ubuntu 16.04 (4.4.0-127) hangs on boot with virtio-scsi MQ enabled
Public bug reported:
We noticed that Ubuntu 16.04 guests running on Nutanix AHV stopped
booting after they were upgraded to the latest kernel (4.4.0-127). Only
guests with scsi mq enabled suffered from this problem. AHV is one of
the few hypervisor products to offer multiqueue for virtio-scsi devices.
Upon further investigation, we could see that the kernel would hang
during the scanning of scsi targets. More specifically, immediately
after coming across a target without any luns present. That's the first
time the kernel destroys a target (given it doesn't have luns). This
could be confirmed with gdb (attached to qemu's gdbserver):
#0 0xc0045039 in ?? ()
#1 0x88022c753c98 in ?? ()
#2 0x815d1de6 in scsi_target_destroy (starget=0x88022ad62400)
at /build/linux-E14mqW/linux-4.4.0/drivers/scsi/scsi_scan.c:322
This shows the guest vCPU stuck on virtio-scsi's implementation of
target_destroy. Despite lacking symbols, we managed to examine the
virtio_scsi_target_state to see that the 'reqs' counter was invalid:
(gdb) p *(struct virtio_scsi_target_state *)starget->hostdata
$6 = {tgt_seq = {sequence = 0}, reqs = {counter = -1}, req_vq =
0x88022cbdd9e8}
(gdb)
This drew our attention to the following patch which is exclusive to the Ubuntu
kernel:
commit f1f609d8015e1d34d39458924dcd9524fccd4307
Author: Jay Vosburgh
Date: Thu Apr 19 21:40:00 2018 +0200
In a nutshell, the patch spins on the target's 'reqs' counter waiting for the
target to quiesce:
--- a/drivers/scsi/virtio_scsi.c
+++ b/drivers/scsi/virtio_scsi.c
@@ -785,6 +785,10 @@ static int virtscsi_target_alloc(struct scsi_target
*starget)
static void virtscsi_target_destroy(struct scsi_target *starget)
{
struct virtio_scsi_target_state *tgt = starget->hostdata;
+
+ /* we can race with concurrent virtscsi_complete_cmd */
+ while (atomic_read(>reqs))
+ cpu_relax();
kfree(tgt);
}
Personally, I think this is a catastrophic way of waiting for a target
to quiesce since virtscsi_target_destroy() is called with IRQs disabled
from scsi_scan.c:scsi_target_destroy(). Devices which take a long time
to quiesce during a target_destroy() could hog the CPU for relatively
long periods of time.
Nevertheless, further study revealed that virtio-scsi itself is broken
in a way that it doesn't increment the 'reqs' counter when submitting
requests on MQ in certain conditions. That caused the counter to go to
-1 (on the completion of the first request) and the CPU to hang
indefinitely.
The following patch fixes the issue:
--- new/linux-4.4.0/drivers/scsi/virtio_scsi.c2018-06-05
10:03:29.083428545 -0700
+++ old/linux-4.4.0/drivers/scsi/virtio_scsi.c2018-06-04
10:23:07.0 -0700
@@ -641,10 +641,9 @@
scsi_target(sc->device)->hostdata;
struct virtio_scsi_vq *req_vq;
- if (shost_use_blk_mq(sh)) {
+ if (shost_use_blk_mq(sh))
req_vq = virtscsi_pick_vq_mq(vscsi, sc);
- atomic_inc(>reqs);
- } else
+ else
req_vq = virtscsi_pick_vq(vscsi, tgt);
return virtscsi_queuecommand(vscsi, req_vq, sc);
Signed-off-by: Felipe Franciosi
Please consider this a urgent fix as all of our customers which use Ubuntu
16.04 and have MQ enabled for better performance will be affected by your
latest update. Our workaround is to recommend that they disable SCSI MQ while
you work on the issue.
Best regards,
Felipe
** Affects: linux (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1775235
Title:
Ubuntu 16.04 (4.4.0-127) hangs on boot with virtio-scsi MQ enabled
Status in linux package in Ubuntu:
New
Bug description:
We noticed that Ubuntu 16.04 guests running on Nutanix AHV stopped
booting after they were upgraded to the latest kernel (4.4.0-127).
Only guests with scsi mq enabled suffered from this problem. AHV is
one of the few hypervisor products to offer multiqueue for virtio-scsi
devices.
Upon further investigation, we could see that the kernel would hang
during the scanning of scsi targets. More specifically, immediately
after coming across a target without any luns present. That's the
first time the kernel destroys a target (given it doesn't have luns).
This could be confirmed with gdb (attached to qemu's gdbserver):
#0 0xc0045039 in ?? ()
#1 0x88022c753c98 in ?? ()
#2 0x815d1de6 in scsi_target_destroy (starget=0x88022ad62400)
at /build/linux-E14mqW/linux-4.4.0/drivers/scsi/scsi_scan.c:322
This shows the guest vCPU stuck on virtio-scsi's implementation of
target_destroy. Despite lacking symbols, we managed to examine the
virtio_scsi_target_state to see that the 'reqs' counter was invalid:
(gdb) p *(struct virtio
[Kernel-packages] [Bug 1775235] Re: Ubuntu 16.04 (4.4.0-127) hangs on boot with virtio-scsi MQ enabled
** Changed in: linux (Ubuntu)
Status: Incomplete => Confirmed
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1775235
Title:
Ubuntu 16.04 (4.4.0-127) hangs on boot with virtio-scsi MQ enabled
Status in linux package in Ubuntu:
Confirmed
Bug description:
We noticed that Ubuntu 16.04 guests running on Nutanix AHV stopped
booting after they were upgraded to the latest kernel (4.4.0-127).
Only guests with scsi mq enabled suffered from this problem. AHV is
one of the few hypervisor products to offer multiqueue for virtio-scsi
devices.
Upon further investigation, we could see that the kernel would hang
during the scanning of scsi targets. More specifically, immediately
after coming across a target without any luns present. That's the
first time the kernel destroys a target (given it doesn't have luns).
This could be confirmed with gdb (attached to qemu's gdbserver):
#0 0xc0045039 in ?? ()
#1 0x88022c753c98 in ?? ()
#2 0x815d1de6 in scsi_target_destroy (starget=0x88022ad62400)
at /build/linux-E14mqW/linux-4.4.0/drivers/scsi/scsi_scan.c:322
This shows the guest vCPU stuck on virtio-scsi's implementation of
target_destroy. Despite lacking symbols, we managed to examine the
virtio_scsi_target_state to see that the 'reqs' counter was invalid:
(gdb) p *(struct virtio_scsi_target_state *)starget->hostdata
$6 = {tgt_seq = {sequence = 0}, reqs = {counter = -1}, req_vq =
0x88022cbdd9e8}
(gdb)
This drew our attention to the following patch which is exclusive to the
Ubuntu kernel:
commit f1f609d8015e1d34d39458924dcd9524fccd4307
Author: Jay Vosburgh
Date: Thu Apr 19 21:40:00 2018 +0200
In a nutshell, the patch spins on the target's 'reqs' counter waiting for the
target to quiesce:
--- a/drivers/scsi/virtio_scsi.c
+++ b/drivers/scsi/virtio_scsi.c
@@ -785,6 +785,10 @@ static int virtscsi_target_alloc(struct scsi_target
*starget)
static void virtscsi_target_destroy(struct scsi_target *starget)
{
struct virtio_scsi_target_state *tgt = starget->hostdata;
+
+ /* we can race with concurrent virtscsi_complete_cmd */
+ while (atomic_read(>reqs))
+ cpu_relax();
kfree(tgt);
}
Personally, I think this is a catastrophic way of waiting for a target
to quiesce since virtscsi_target_destroy() is called with IRQs
disabled from scsi_scan.c:scsi_target_destroy(). Devices which take a
long time to quiesce during a target_destroy() could hog the CPU for
relatively long periods of time.
Nevertheless, further study revealed that virtio-scsi itself is broken
in a way that it doesn't increment the 'reqs' counter when submitting
requests on MQ in certain conditions. That caused the counter to go to
-1 (on the completion of the first request) and the CPU to hang
indefinitely.
The following patch fixes the issue:
--- new/linux-4.4.0/drivers/scsi/virtio_scsi.c2018-06-05
10:03:29.083428545 -0700
+++ old/linux-4.4.0/drivers/scsi/virtio_scsi.c2018-06-04
10:23:07.0 -0700
@@ -641,10 +641,9 @@
scsi_target(sc->device)->hostdata;
struct virtio_scsi_vq *req_vq;
- if (shost_use_blk_mq(sh)) {
+ if (shost_use_blk_mq(sh))
req_vq = virtscsi_pick_vq_mq(vscsi, sc);
- atomic_inc(>reqs);
- } else
+ else
req_vq = virtscsi_pick_vq(vscsi, tgt);
return virtscsi_queuecommand(vscsi, req_vq, sc);
Signed-off-by: Felipe Franciosi
Please consider this a urgent fix as all of our customers which use Ubuntu
16.04 and have MQ enabled for better performance will be affected by your
latest update. Our workaround is to recommend that they disable SCSI MQ while
you work on the issue.
Best regards,
Felipe
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1775235/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
