[Bug 1907421] Re: [UBUNTU 21.04] vfio: pass DMA availability information to userspace
Patch "vfio iommu: Add dma available capability" now landed as 6507174efc67 in focal-updates and is part of Ubuntu-5.4.0-67. Hence updating the 'affects focal' entry as Fix Released and with that closing the entire ticket. ** Changed in: linux (Ubuntu Focal) Status: Fix Committed => Fix Released ** Changed in: ubuntu-z-systems Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907421 Title: [UBUNTU 21.04] vfio: pass DMA availability information to userspace To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1907421/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907421] Re: [UBUNTU 21.04] vfio: pass DMA availability information to userspace
Commit 6507174efc67 "vfio iommu: Add dma available capability" comes with Ubuntu-5.4.0-67. It's still in proposed as of today. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907421 Title: [UBUNTU 21.04] vfio: pass DMA availability information to userspace To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1907421/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907421] Re: [UBUNTU 21.04] vfio: pass DMA availability information to userspace
For groovy the patches are in Ubuntu-5.8.0-44.50 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907421 Title: [UBUNTU 21.04] vfio: pass DMA availability information to userspace To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1907421/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907421] Re: [UBUNTU 21.04] vfio: pass DMA availability information to userspace
Now that kernel 5.10 landed in hirsute's release pocket: linux-generic | 5.10.0.14.16 | hirsute the 'hirsute' part can be updated to 'Fix Released". ** Changed in: linux (Ubuntu Hirsute) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907421 Title: [UBUNTU 21.04] vfio: pass DMA availability information to userspace To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1907421/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907421] Re: [UBUNTU 21.04] vfio: pass DMA availability information to userspace
** Changed in: linux (Ubuntu Focal) Status: In Progress => Fix Committed ** Changed in: ubuntu-z-systems Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907421 Title: [UBUNTU 21.04] vfio: pass DMA availability information to userspace To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1907421/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907421] Re: [UBUNTU 21.04] vfio: pass DMA availability information to userspace
The fix is currently prepared for focal based on LP 1913487 - "Focal update: v5.4.90 upstream stable release" https://bugs.launchpad.net/bugs/1913487 Hence aligning the status of the focal entry of this bug to LP1913487 and updating the status to In Progress. ** Changed in: linux (Ubuntu Focal) Status: Confirmed => In Progress ** Changed in: ubuntu-z-systems Status: Confirmed => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907421 Title: [UBUNTU 21.04] vfio: pass DMA availability information to userspace To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1907421/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907421] Re: [UBUNTU 21.04] vfio: pass DMA availability information to userspace
Thx, I'm adjusting the tag accordingly ... ** Tags removed: verification-needed-groovy ** Tags added: verification-done-groovy -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907421 Title: [UBUNTU 21.04] vfio: pass DMA availability information to userspace To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1907421/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907421] Re: [UBUNTU 21.04] vfio: pass DMA availability information to userspace
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed- groovy' to 'verification-done-groovy'. If the problem still exists, change the tag 'verification-needed-groovy' to 'verification-failed- groovy'. If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you! ** Tags added: verification-needed-groovy -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907421 Title: [UBUNTU 21.04] vfio: pass DMA availability information to userspace To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1907421/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907421] Re: [UBUNTU 21.04] vfio: pass DMA availability information to userspace
** Changed in: linux (Ubuntu Groovy) Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907421 Title: [UBUNTU 21.04] vfio: pass DMA availability information to userspace To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1907421/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907421] Re: [UBUNTU 21.04] vfio: pass DMA availability information to userspace
Kernel SRU request submitted for groovy: https://lists.ubuntu.com/archives/kernel-team/2021-January/thread.html#116618 changing status to 'In Progress'. ** Description changed: Description: vfio: pass DMA availability information to userspace Symptom: vfio-pci device on s390 enters error state Problem: Commit 492855939bdb added a limit to the number of concurrent -DMA requests for a vfio container. However, lazy unmapping in -s390 can in fact cause quite a large number of outstanding DMA -requests to build up prior to being purged, potentially the -entire guest DMA space. This results in unexpected errors seen -in qemu such as 'VFIO_MAP_DMA failed: No space left on device' + DMA requests for a vfio container. However, lazy unmapping in + s390 can in fact cause quite a large number of outstanding DMA + requests to build up prior to being purged, potentially the + entire guest DMA space. This results in unexpected errors seen + in qemu such as 'VFIO_MAP_DMA failed: No space left on device' Solution: The solution requires a change to both kernel and qemu - For -the kernel, add the ability to provide the number of allowable -DMA requests via the VFIO_IOMMU_GET_INFO ioctl. + the kernel, add the ability to provide the number of allowable + DMA requests via the VFIO_IOMMU_GET_INFO ioctl. Reproduction: Put a vfio-pci device on s390 under I/O load Upstream-ID: a717072007e8aedd3f951726d8cf55454860b30d -7d6e1329652ed971d1b6e0e7bea66fba5044e271 + 7d6e1329652ed971d1b6e0e7bea66fba5044e271 Need also to be integrated into 20.10 and 20.04. - OK, just to clarify we don't need to fix bionic for this one, but rather - focal (20.04) and groovy (20.10). Furthermore, for 20.04, 20.10 and - 21.04 ONLY commit 7d6e1329652ed971d1b6e0e7bea66fba5044e271 is needed, - the other was a pre-req that is already present. + OK, just to clarify we don't need to fix bionic for this one, but rather focal (20.04) and groovy (20.10). Furthermore, for 20.04, 20.10 and 21.04 ONLY commit 7d6e1329652ed971d1b6e0e7bea66fba5044e271 is needed, the other was a pre-req that is already present. + __ + + SRU Justification: + + [Impact] + + * In case a vfio-pci device on s390x is under I/O load, vfio-pci device + may end up in error state. + + * The commit 492855939bdb added a limit to the number of concurrent DMA + requests for a vfio container. + + * However, lazy unmapping in s390x can in fact cause quite a large number of outstanding DMA requests + to build up prior to being purged - potentially the entire guest DMA space. + + * This results in unexpected errors seen in qemu such as 'VFIO_MAP_DMA + failed: No space left on device'. + + * The solution requires a change to both kernel and qemu. + + * The kernel side of things is addressed by this SRU. + + * The fix adds the ability to provide the number of allowable DMA + requests via VFIO_IOMMU_GET_INFO ioctl. + + * The actual fix comes with commit 7d6e1329652e, but another fix + ccd59dce1a21 is needed to get it cleanly applied. + + [Fix] + + * ccd59dce1a21f473518bf273bdf5b182bab955b3 ccd59dce1a21 "vfio/type1: + Refactor vfio_iommu_type1_ioctl()" + + * 7d6e1329652ed971d1b6e0e7bea66fba5044e271 7d6e1329652e "vfio iommu: Add + dma available capability" + + [Test Case] + + * IBM Z or LinuxONE hardware with Ubuntu Server 20.10 installed. + + * PCIe adapters in place that provide vfio, like RoCE Express 2. + + * A KVM host needs to be setup and a KVM guest (use again 20.10) that + uses vfio. + + * Generate I/O that flows through the vf and watch out for error like + 'VFIO_MAP_DMA failed: No space left on device' in the log. + + [Regression Potential] + + * The first patch ccd59dce1a21 modifies the common code file + drivers/vfio/vfio_iommu_type1.c quity significantly. + + * But the reason is not that it introduces a lot of new things, it's a + refactoring patch. + + * Nevertheless if done in a bad way it can significantly harm the IO + memory management of virtual function adapters. + + * In worst case it may break them entirely, instead of 'just' exeeding + the entire DMA space. + + * Things could also go wrong while doing the mapping and unmapping of + DMA, that may even have an impact beyond vf adapters - harming other DMA + devices. + + * The handling of dirty pages is also touched and the ioctl itself - + which is important to keep the control of the devices. + + * But as said before, it re-factoring work, it's upstream accepted since + 5.9 and the provenance shows that many engineers had an eye on these + changes. + + * The second patch 7d6e1329652e - that inclides the needed fix - comes + with far less modifications. + + * It also tounches drivers/vfio/vfio_iommu_type1.c
[Bug 1907421] Re: [UBUNTU 21.04] vfio: pass DMA availability information to userspace
A PPA was created with a patched groovy/20.10 kernel for further testing: https://people.canonical.com/~fheimes/lp1907421 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907421 Title: [UBUNTU 21.04] vfio: pass DMA availability information to userspace To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1907421/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907421] Re: [UBUNTU 21.04] vfio: pass DMA availability information to userspace
Regarding comment #6: (Didn't noticed that the 5.9 series is also closed now - nevermind ...) The commit ccd59dce1a21 is indeed the missing puzzle piece. I was just surprised that just one commit fixes the significant delta, but since it contains a huge chunk of refactoring work: "vfio/type1: Refactor vfio_iommu_type1_ioctl()" this explains it. I just tried to cherry pick this one (ccd59dce1a21) and the initial one on top (7d6e1329652e) and everything applied cleanly. With that I can do a manual kernel SRU to groovy, so that we can fix the potential upgrade regression situation. Thanks for letting me know the missing one ... And reading comment #7 I think we can (well, actually need) to ask you for the verification tests once the kernel fixes are in, since you have the right hw and a custom qemu. ** Changed in: linux (Ubuntu Groovy) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907421 Title: [UBUNTU 21.04] vfio: pass DMA availability information to userspace To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1907421/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907421] Re: [UBUNTU 21.04] vfio: pass DMA availability information to userspace
Hi, You also mentioned qemu in the description, I think what is related (by title and it has your name) is the following series + one fixup: 77280d33b s390x: fix build for --without-default-devices 37fa32de70 s390x/pci: Honor DMA limits set by vfio cd7498d07f s390x/pci: Add routine to get the vfio dma available count 7486a62845 vfio: Find DMA available capability 3ab7a0b40d vfio: Create shared routine for scanning info capabilities 408b55db8b s390x/pci: Move header files to include/hw/s390x 53ba2eee52 linux-headers: update against 5.10-rc1 84567ea763 update-linux-headers: Add vfio_zdev.h That is in qemu 5.2 - and if that "is it" then in Hirsute things are already fixed by now. But that is quite a list and for backports we will need to reduce that to the minimum that is needed. Did you check if you can (or have even a branch somewhere) backport these on top of 4.2 and/or 5.0 ? Also if this shall be an SRU we would need a testcase, is there anything that can be isolated as test instructions? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907421 Title: [UBUNTU 21.04] vfio: pass DMA availability information to userspace To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1907421/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907421] Re: [UBUNTU 21.04] vfio: pass DMA availability information to userspace
Hi Matthew, I'm updating the hirsute entry to Fix Committed, since the 5.10 kernel is in hirsute-proposed and the requested patch is included: Ubuntu-5.10.0-12.13 v5.10 So we are good with 21.04/hirsute. And it's great that see that you've managed to get it accepted via upstream stable, since this provides a lot of confidence about the fix that we need for an LTS SRU - and since we have a process in place to pick up stable (upstream) updates, that will find (semi-automatically) it's way into 20.04/focal's Ubuntu kernel 5.4 soon. So this is highly appreciated. However, to avoid any regression for people who upgrade from an Ubuntu release n to n+1 (in this case 20.04/focal to 20.10/groovy) one requirement of our SRU process is to get the patches in to the Ubuntu releases/kernels starting from newest to the oldest Ubuntu release the patch needs to be applied to. Hence me trying to get the patch applied to the groovy master-next tree, based on kernel 5.8, which failed (LP comment #2) (since we knew it's already in 5.10 which is the target for 21.04). Well, I can retry to apply the patch to groovy master-next, since the groovy master-next tree evolved over time, but I guess it still does not apply - hence me asking for a backport (#2) Btw. even if 5.8.y ran out of upstream stable support, we still look at the 5.9.y stable updates for inclusion into 5.8. So tagging the fix for stable 5.9.y would be an option, too. ** Changed in: linux (Ubuntu Hirsute) Status: In Progress => Fix Committed ** Changed in: linux (Ubuntu Focal) Status: Incomplete => Confirmed ** Changed in: ubuntu-z-systems Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907421 Title: [UBUNTU 21.04] vfio: pass DMA availability information to userspace To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1907421/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907421] Re: [UBUNTU 21.04] vfio: pass DMA availability information to userspace
Trying to cherry-pick 7d6e1329652e from focal master-next or groovy master-next ends up with a significant merge conflict. I guess proper backports are needed and should certainly be addressed via upstream stable, since it's all common code and that would address the maintenance burden for the changes and lower the risk. ** Changed in: linux (Ubuntu Focal) Status: New => Incomplete ** Changed in: linux (Ubuntu Groovy) Status: New => Incomplete ** Changed in: ubuntu-z-systems Status: Triaged => Incomplete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907421 Title: [UBUNTU 21.04] vfio: pass DMA availability information to userspace To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1907421/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907421] Re: [UBUNTU 21.04] vfio: pass DMA availability information to userspace
The patch is in Ubuntu UNSTABLE (that will likely become the kernel for hirsute) as: 7d6e1329652e "vfio iommu: Add dma available capability" starting with: Ubuntu-5.10-5.10.0-0.1 Hence updating Hirsute entry to 'In Progress'. ** Changed in: linux (Ubuntu Hirsute) Status: New => In Progress ** Changed in: ubuntu-z-systems Assignee: (unassigned) => Skipper Bug Screeners (skipper-screen-team) ** Changed in: linux (Ubuntu Hirsute) Assignee: Skipper Bug Screeners (skipper-screen-team) => (unassigned) ** Changed in: linux (Ubuntu Groovy) Assignee: (unassigned) => Frank Heimes (fheimes) ** Changed in: linux (Ubuntu Focal) Assignee: (unassigned) => Frank Heimes (fheimes) ** Changed in: ubuntu-z-systems Status: New => Triaged -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907421 Title: [UBUNTU 21.04] vfio: pass DMA availability information to userspace To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1907421/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1907421] Re: [UBUNTU 21.04] vfio: pass DMA availability information to userspace
** Also affects: linux (Ubuntu Hirsute) Importance: Undecided Assignee: Skipper Bug Screeners (skipper-screen-team) Status: New ** Also affects: linux (Ubuntu Groovy) Importance: Undecided Status: New ** Also affects: linux (Ubuntu Focal) Importance: Undecided Status: New ** Also affects: ubuntu-z-systems Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1907421 Title: [UBUNTU 21.04] vfio: pass DMA availability information to userspace To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1907421/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
