[Kernel-packages] [Bug 1917203] Re: AMD-Vi: Unable to read/write to IOMMU perf counter

[Alex Hung]

** Changed in: linux (Ubuntu)
 Assignee: Alex Hung (alexhung) => (unassigned)

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1917203

Title:
  AMD-Vi: Unable to read/write to IOMMU perf counter

Status in linux package in Ubuntu:
  In Progress

Bug description:
  This boot warning (concealed by grub not taking over the fbcon console
  on a solo installation but always present on a multi-boot) has been
  bothering Linux users of AMD Ryzen machines for some time.

  The problem is currently under discussion at kernel level
  .

  One solution proposed by Suravee Suthikulpanit

  works but inserts a boot-up delay of (at least) 100 msec. A second
  option by Alexander Monakov 
   also works but
  inserts no delay and more-or-less just moves one line of code.

  I've tried both solutions with kernel rebuilds for both Breezy kernel
  5.8.18 and Hirsute kernel 5.10.11 and both work on my AMD Ryzen 2400G.
  Could I encourage your kernel experts to evaluate the situation (I
  think SuSE folk already are).

  My suggestion (as a humble user :-) ) would be to fold Alex's simple
  patch into your own list of kernel tweeks and get the correction out
  with the upcoming Hirsute release.

  Best regards and all respect!

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1917203/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1917203] Re: AMD-Vi: Unable to read/write to IOMMU perf counter

[Alex Hung]

@david, I am also watching the bugzilla 201753. I am also looking
forward seeing a better solution

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1917203

Title:
  AMD-Vi: Unable to read/write to IOMMU perf counter

Status in linux package in Ubuntu:
  In Progress

Bug description:
  This boot warning (concealed by grub not taking over the fbcon console
  on a solo installation but always present on a multi-boot) has been
  bothering Linux users of AMD Ryzen machines for some time.

  The problem is currently under discussion at kernel level
  .

  One solution proposed by Suravee Suthikulpanit

  works but inserts a boot-up delay of (at least) 100 msec. A second
  option by Alexander Monakov 
   also works but
  inserts no delay and more-or-less just moves one line of code.

  I've tried both solutions with kernel rebuilds for both Breezy kernel
  5.8.18 and Hirsute kernel 5.10.11 and both work on my AMD Ryzen 2400G.
  Could I encourage your kernel experts to evaluate the situation (I
  think SuSE folk already are).

  My suggestion (as a humble user :-) ) would be to fold Alex's simple
  patch into your own list of kernel tweeks and get the correction out
  with the upcoming Hirsute release.

  Best regards and all respect!

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1917203/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1917203] Re: AMD-Vi: Unable to read/write to IOMMU perf counter

[David Coe]

Hi Alex!

Herewith a summary of our accumulated results [1] for Surajee's IOMMU
patch using 20 msecs wait and logged retries on AMD's Ryzen:

Ryzen KernelColdWarm

4700U   5.11.0-116  1
3500U   5.11.7   5  6
2500U   5.8.0-455
5.12.0 RC3   5> 5
2400G   5.11.0-116  6
5.8.0-45 5  6
2200G   ? >10

Two points are clear:

1. there are differences between cold and warm boot, mostly marginal but
marked and very consistent with the quite new 4700U.

2. the choice of 5 as the maximum retry number is unfortunate. Mostly,
it guarantees all our Ryzens just fail the IOMMU write test!

The 2200G is a bit of a odd-ball and it's owner, understandably, wants a
more elegant solution than just upping the number of retries. For the
rest of us, a maximum of 6 or 7 would sort it.

Best regards and many thanks

David

[1] https://bugzilla.kernel.org/show_bug.cgi?id=201753

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1917203

Title:
  AMD-Vi: Unable to read/write to IOMMU perf counter

Status in linux package in Ubuntu:
  In Progress

Bug description:
  This boot warning (concealed by grub not taking over the fbcon console
  on a solo installation but always present on a multi-boot) has been
  bothering Linux users of AMD Ryzen machines for some time.

  The problem is currently under discussion at kernel level
  .

  One solution proposed by Suravee Suthikulpanit

  works but inserts a boot-up delay of (at least) 100 msec. A second
  option by Alexander Monakov 
   also works but
  inserts no delay and more-or-less just moves one line of code.

  I've tried both solutions with kernel rebuilds for both Breezy kernel
  5.8.18 and Hirsute kernel 5.10.11 and both work on my AMD Ryzen 2400G.
  Could I encourage your kernel experts to evaluate the situation (I
  think SuSE folk already are).

  My suggestion (as a humble user :-) ) would be to fold Alex's simple
  patch into your own list of kernel tweeks and get the correction out
  with the upcoming Hirsute release.

  Best regards and all respect!

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1917203/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1917203] Re: AMD-Vi: Unable to read/write to IOMMU perf counter

[David Coe]

I've just tried Surajee's patch on Ubuntu's latest kernel 5.11.0-11 for
upcoming Hirsute and it consistently unlatches IOMMU after 6 x 20 msec
tries on my Ryzen 2400G. Mainline 5.12-rc2 gave identical results.

Paul Menzel too gets failure even after 10 x 20 msecs on a 2200G and
regards so long a delay as most unacceptable. Alexander Monakov's very
simple patch always works for me but I haven't seen data from other
people. Surajee has mentioned problems with it for some part of AMD's
product range and, as one of AMD's IOMMU experts down at Austen Texas,
he should know his onions :-).

I would be tempted to run with the patch (11 x 20 msecs), log the exit
count number and get some user feedback. Hopefully an improved patch
will emerge! AMD may have commercial as well as technical reasons not
leave it unaddressed.

Best regards

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1917203

Title:
  AMD-Vi: Unable to read/write to IOMMU perf counter

Status in linux package in Ubuntu:
  In Progress

Bug description:
  This boot warning (concealed by grub not taking over the fbcon console
  on a solo installation but always present on a multi-boot) has been
  bothering Linux users of AMD Ryzen machines for some time.

  The problem is currently under discussion at kernel level
  .

  One solution proposed by Suravee Suthikulpanit

  works but inserts a boot-up delay of (at least) 100 msec. A second
  option by Alexander Monakov 
   also works but
  inserts no delay and more-or-less just moves one line of code.

  I've tried both solutions with kernel rebuilds for both Breezy kernel
  5.8.18 and Hirsute kernel 5.10.11 and both work on my AMD Ryzen 2400G.
  Could I encourage your kernel experts to evaluate the situation (I
  think SuSE folk already are).

  My suggestion (as a humble user :-) ) would be to fold Alex's simple
  patch into your own list of kernel tweeks and get the correction out
  with the upcoming Hirsute release.

  Best regards and all respect!

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1917203/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1917203] Re: AMD-Vi: Unable to read/write to IOMMU perf counter

[Alex Hung]

I also tried to increase the msleep, (retry = 5)
-   msleep(20);
+   msleep(40);

The same result on #7. I am not sure whether this applies to my AMD
Ryzen 5 2500U.

Update: If the system is powered off for long time (i.e. 5+ mins), the
patch will work.

In any case, it seems there are still room for improvement.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1917203

Title:
  AMD-Vi: Unable to read/write to IOMMU perf counter

Status in linux package in Ubuntu:
  In Progress

Bug description:
  This boot warning (concealed by grub not taking over the fbcon console
  on a solo installation but always present on a multi-boot) has been
  bothering Linux users of AMD Ryzen machines for some time.

  The problem is currently under discussion at kernel level
  .

  One solution proposed by Suravee Suthikulpanit

  works but inserts a boot-up delay of (at least) 100 msec. A second
  option by Alexander Monakov 
   also works but
  inserts no delay and more-or-less just moves one line of code.

  I've tried both solutions with kernel rebuilds for both Breezy kernel
  5.8.18 and Hirsute kernel 5.10.11 and both work on my AMD Ryzen 2400G.
  Could I encourage your kernel experts to evaluate the situation (I
  think SuSE folk already are).

  My suggestion (as a humble user :-) ) would be to fold Alex's simple
  patch into your own list of kernel tweeks and get the correction out
  with the upcoming Hirsute release.

  Best regards and all respect!

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1917203/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1917203] Re: AMD-Vi: Unable to read/write to IOMMU perf counter

[Alex Hung]

I did some tests with retry, based on latest mainline kernel today

1. change retry to 10
2. install kernel and reboot/poweroff
3. on the first boot after install, regarding of reboot or poweroff, the patch 
works
4. after second boot (reboot or poweroff), it always fails

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1917203

Title:
  AMD-Vi: Unable to read/write to IOMMU perf counter

Status in linux package in Ubuntu:
  In Progress

Bug description:
  This boot warning (concealed by grub not taking over the fbcon console
  on a solo installation but always present on a multi-boot) has been
  bothering Linux users of AMD Ryzen machines for some time.

  The problem is currently under discussion at kernel level
  .

  One solution proposed by Suravee Suthikulpanit

  works but inserts a boot-up delay of (at least) 100 msec. A second
  option by Alexander Monakov 
   also works but
  inserts no delay and more-or-less just moves one line of code.

  I've tried both solutions with kernel rebuilds for both Breezy kernel
  5.8.18 and Hirsute kernel 5.10.11 and both work on my AMD Ryzen 2400G.
  Could I encourage your kernel experts to evaluate the situation (I
  think SuSE folk already are).

  My suggestion (as a humble user :-) ) would be to fold Alex's simple
  patch into your own list of kernel tweeks and get the correction out
  with the upcoming Hirsute release.

  Best regards and all respect!

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1917203/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1917203] Re: AMD-Vi: Unable to read/write to IOMMU perf counter

[David Coe]

Hi Alex!

There are further results [1] for Suravee's patch on Ryzen 2500U
(BubuXP) and 2400G (myself). Both show a slight difference between cold
boot (retry 5 x 20 msecs = success) and warm boot (retry 6 x 20 msecs =
failure).

It would be worth (even slightly) increasing the number of retries
should you backport this commit <6778ff5b21bd8e78> to hirsute or breezy.

Regards

[1] https://bugzilla.kernel.org/show_bug.cgi?id=201753

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1917203

Title:
  AMD-Vi: Unable to read/write to IOMMU perf counter

Status in linux package in Ubuntu:
  In Progress

Bug description:
  This boot warning (concealed by grub not taking over the fbcon console
  on a solo installation but always present on a multi-boot) has been
  bothering Linux users of AMD Ryzen machines for some time.

  The problem is currently under discussion at kernel level
  .

  One solution proposed by Suravee Suthikulpanit

  works but inserts a boot-up delay of (at least) 100 msec. A second
  option by Alexander Monakov 
   also works but
  inserts no delay and more-or-less just moves one line of code.

  I've tried both solutions with kernel rebuilds for both Breezy kernel
  5.8.18 and Hirsute kernel 5.10.11 and both work on my AMD Ryzen 2400G.
  Could I encourage your kernel experts to evaluate the situation (I
  think SuSE folk already are).

  My suggestion (as a humble user :-) ) would be to fold Alex's simple
  patch into your own list of kernel tweeks and get the correction out
  with the upcoming Hirsute release.

  Best regards and all respect!

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1917203/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1917203] Re: AMD-Vi: Unable to read/write to IOMMU perf counter

[David Coe]

Hi Alex!

A splendidly prompt piece of cherry-picking. Many thanks.

I've checked your test release on my Ryzen 2400G on current Ubuntu 20.10 and it 
still gives the diagostic:
  AMD-Vi: Unable to read/write to IOMMU perf counter

It's to be expected! The little crew engaged with testing Suravee
Suthikulpanit's patch [1] are finding that the 2400G (that's me) needs
120 msec for power-gating and the 2200G (that's Paul Menzel] needs >200
msecs. Suravee's RFC v3 provides 5 x 20 msecs delay which just isn't
enough for these lesser Ryzen's :-).

If you change Suravee's patch line

   for (retry = 5; retry; retry--) {
to
   for (retry = 25; retry; retry--) {

all will be well (imho). The maximum delay is indeed outrageous but only
that really necessary to enable IOMMU is actually used. Eventually a
proper solution will be found (in firmware or silicon) but (unlike at
present) the many users of entry-level Ryzen CPU's will not have their
IOMMU crippled during linux boot-up.

Incidentally, if you do rebuild your (very kind) test-release with the
above change (please, please), could you also include the deb package
for linux-tools for the same kernel. Things like the command "perf stat
-a -e amd_iommu/mem_trans_total/ test" (used to test IOMMU performance
in user-space) are tied to the running kernel-version and will then
work.

Best regards and again many thanks

David

[1] https://bugzilla.kernel.org/show_bug.cgi?id=201753

** Bug watch added: Linux Kernel Bug Tracker #201753
   https://bugzilla.kernel.org/show_bug.cgi?id=201753

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1917203

Title:
  AMD-Vi: Unable to read/write to IOMMU perf counter

Status in linux package in Ubuntu:
  In Progress

Bug description:
  This boot warning (concealed by grub not taking over the fbcon console
  on a solo installation but always present on a multi-boot) has been
  bothering Linux users of AMD Ryzen machines for some time.

  The problem is currently under discussion at kernel level
  .

  One solution proposed by Suravee Suthikulpanit

  works but inserts a boot-up delay of (at least) 100 msec. A second
  option by Alexander Monakov 
   also works but
  inserts no delay and more-or-less just moves one line of code.

  I've tried both solutions with kernel rebuilds for both Breezy kernel
  5.8.18 and Hirsute kernel 5.10.11 and both work on my AMD Ryzen 2400G.
  Could I encourage your kernel experts to evaluate the situation (I
  think SuSE folk already are).

  My suggestion (as a humble user :-) ) would be to fold Alex's simple
  patch into your own list of kernel tweeks and get the correction out
  with the upcoming Hirsute release.

  Best regards and all respect!

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1917203/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1917203] Re: AMD-Vi: Unable to read/write to IOMMU perf counter

[Alex Hung]

@david,

I cherry-picked 6778ff5b21bd8e78 and built a test kernel @
https://people.canonical.com/~alexhung/LP1917203/. I tested on an AMD
Ryzen 5 2500U

Before
[0.975812] pci :00:00.2: AMD-Vi: Unable to read/write to IOMMU perf 
counter.

After:
[1.006639] pci :00:00.2: AMD-Vi: IOMMU performance counters supported

It would be great if you can give it a try and update the results.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1917203

Title:
  AMD-Vi: Unable to read/write to IOMMU perf counter

Status in linux package in Ubuntu:
  In Progress

Bug description:
  This boot warning (concealed by grub not taking over the fbcon console
  on a solo installation but always present on a multi-boot) has been
  bothering Linux users of AMD Ryzen machines for some time.

  The problem is currently under discussion at kernel level
  .

  One solution proposed by Suravee Suthikulpanit

  works but inserts a boot-up delay of (at least) 100 msec. A second
  option by Alexander Monakov 
   also works but
  inserts no delay and more-or-less just moves one line of code.

  I've tried both solutions with kernel rebuilds for both Breezy kernel
  5.8.18 and Hirsute kernel 5.10.11 and both work on my AMD Ryzen 2400G.
  Could I encourage your kernel experts to evaluate the situation (I
  think SuSE folk already are).

  My suggestion (as a humble user :-) ) would be to fold Alex's simple
  patch into your own list of kernel tweeks and get the correction out
  with the upcoming Hirsute release.

  Best regards and all respect!

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1917203/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1917203] Re: AMD-Vi: Unable to read/write to IOMMU perf counter

[Alex Hung]

Suravee's patch (commit 6778ff5b21bd8e78) is now in mainline kernel. It
is more appropriate to backport it to Ubuntu kernels.

Let's wait for hirsute to rebase to 5.11 and send a SRU.

** Changed in: linux (Ubuntu)
 Assignee: (unassigned) => Alex Hung (alexhung)

** Changed in: linux (Ubuntu)
   Status: Incomplete => In Progress

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1917203

Title:
  AMD-Vi: Unable to read/write to IOMMU perf counter

Status in linux package in Ubuntu:
  In Progress

Bug description:
  This boot warning (concealed by grub not taking over the fbcon console
  on a solo installation but always present on a multi-boot) has been
  bothering Linux users of AMD Ryzen machines for some time.

  The problem is currently under discussion at kernel level
  .

  One solution proposed by Suravee Suthikulpanit

  works but inserts a boot-up delay of (at least) 100 msec. A second
  option by Alexander Monakov 
   also works but
  inserts no delay and more-or-less just moves one line of code.

  I've tried both solutions with kernel rebuilds for both Breezy kernel
  5.8.18 and Hirsute kernel 5.10.11 and both work on my AMD Ryzen 2400G.
  Could I encourage your kernel experts to evaluate the situation (I
  think SuSE folk already are).

  My suggestion (as a humble user :-) ) would be to fold Alex's simple
  patch into your own list of kernel tweeks and get the correction out
  with the upcoming Hirsute release.

  Best regards and all respect!

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1917203/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1917203] Re: AMD-Vi: Unable to read/write to IOMMU perf counter

[Steve Langasek]

The ubuntu-cdimage project is for tracking bugs in the code used to
create the install media for Ubuntu. Please use the ubuntu-bug command
to file bug reports against the individual packages you are finding bugs
in.

** Project changed: ubuntu-cdimage => linux (Ubuntu)

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1917203

Title:
  AMD-Vi: Unable to read/write to IOMMU perf counter

Status in linux package in Ubuntu:
  Incomplete

Bug description:
  This boot warning (concealed by grub not taking over the fbcon console
  on a solo installation but always present on a multi-boot) has been
  bothering Linux users of AMD Ryzen machines for some time.

  The problem is currently under discussion at kernel level
  .

  One solution proposed by Suravee Suthikulpanit

  works but inserts a boot-up delay of (at least) 100 msec. A second
  option by Alexander Monakov 
   also works but
  inserts no delay and more-or-less just moves one line of code.

  I've tried both solutions with kernel rebuilds for both Breezy kernel
  5.8.18 and Hirsute kernel 5.10.11 and both work on my AMD Ryzen 2400G.
  Could I encourage your kernel experts to evaluate the situation (I
  think SuSE folk already are).

  My suggestion (as a humble user :-) ) would be to fold Alex's simple
  patch into your own list of kernel tweeks and get the correction out
  with the upcoming Hirsute release.

  Best regards and all respect!

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1917203/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp