Re: 6.9/amd64 runaway acpi process on Thinkpad T580

[Theo de Raadt]

Mike Larkin  wrote:

> On Wed, Sep 29, 2021 at 08:44:54PM -0400, David Anthony wrote:
> > After enabling "BIOS Thunderbolt Assist", I experience consistent machine
> > slowdown on my T480. Previously, I experienced slowdown after power cycling
> > my machine occasionally. Currently, with this BIOS setting enabled, I
> > experience slowdown consistently.
> >
> > I am sorry but I don't know enough technically as to discern why. I am
> > simply reporting my user experience. I will re-disable the Thunderbolt
> > assist for now.
> >
> 
> If someone would build an ACPI_DEBUG kernel and show us what GPE is stuck
> then we can make forward progress (we need an acpidump of that machine
> also).
> 
> Otherwise, its like throwing darts in the dark.

Or, someone with a machine which has the problem can give it to Mike,
or a few other developers who understand this problem area.

I'm not joking.  Give it.  It would be a public service.

Re: 6.9/amd64 runaway acpi process on Thinkpad T580

[Daniel Wilkins]

On Wed, Sep 29, 2021 at 06:29:08PM -0700, Mike Larkin wrote:
> On Wed, Sep 29, 2021 at 08:44:54PM -0400, David Anthony wrote:
> > After enabling "BIOS Thunderbolt Assist", I experience consistent machine
> > slowdown on my T480. Previously, I experienced slowdown after power cycling
> > my machine occasionally. Currently, with this BIOS setting enabled, I
> > experience slowdown consistently.
> >
> > I am sorry but I don't know enough technically as to discern why. I am
> > simply reporting my user experience. I will re-disable the Thunderbolt
> > assist for now.
> >
>
> If someone would build an ACPI_DEBUG kernel and show us what GPE is stuck
> then we can make forward progress (we need an acpidump of that machine
> also).
>
> Otherwise, its like throwing darts in the dark.
>
> -ml

I could give it a shot. Do you want all three possible states for the
dumps? (disabled, working. Disabled, looped acpi0. Enabled, working.)

It probably won't be until tomorrow since it's already pretty late,
though.

Danny

Re: 6.9/amd64 runaway acpi process on Thinkpad T580

[Mike Larkin]

On Wed, Sep 29, 2021 at 08:44:54PM -0400, David Anthony wrote:
> After enabling "BIOS Thunderbolt Assist", I experience consistent machine
> slowdown on my T480. Previously, I experienced slowdown after power cycling
> my machine occasionally. Currently, with this BIOS setting enabled, I
> experience slowdown consistently.
>
> I am sorry but I don't know enough technically as to discern why. I am
> simply reporting my user experience. I will re-disable the Thunderbolt
> assist for now.
>

If someone would build an ACPI_DEBUG kernel and show us what GPE is stuck
then we can make forward progress (we need an acpidump of that machine
also).

Otherwise, its like throwing darts in the dark.

-ml

> On 9/29/21 2:58 PM, David Anthony wrote:
> > Another T480 user who has noticed the same problem. Per advice given,
> > I've just enabled "BIOS Thunderbolt Assist". I will report back if I
> > notice the problem persists.
> >
> > On 9/19/21 4:50 AM, Daniel Wilkins wrote:
> > > I've ran into this on my T480, it seems most consistently triggered
> > > by power
> > > cycles caused by running out of battery. The bug's existed for quite
> > > a few
> > > years (I think I first noticed it in 2019.) If I recall correctly I've
> > > posted it to the list a couple of times but I don't think any
> > > concrete answers
> > > ever emerged; your report is more thorough than mine were though.
> > > I do remember that it never happened on my T430, but that's quite the
> > > hardware gap.
> > >
> >
>

Re: 6.9/amd64 runaway acpi process on Thinkpad T580

[David Anthony]

After enabling "BIOS Thunderbolt Assist", I experience consistent 
machine slowdown on my T480. Previously, I experienced slowdown after 
power cycling my machine occasionally. Currently, with this BIOS setting 
enabled, I experience slowdown consistently.


I am sorry but I don't know enough technically as to discern why. I am 
simply reporting my user experience. I will re-disable the Thunderbolt 
assist for now.


On 9/29/21 2:58 PM, David Anthony wrote:
Another T480 user who has noticed the same problem. Per advice given, 
I've just enabled "BIOS Thunderbolt Assist". I will report back if I 
notice the problem persists.


On 9/19/21 4:50 AM, Daniel Wilkins wrote:
I've ran into this on my T480, it seems most consistently triggered 
by power
cycles caused by running out of battery. The bug's existed for quite 
a few

years (I think I first noticed it in 2019.) If I recall correctly I've
posted it to the list a couple of times but I don't think any 
concrete answers

ever emerged; your report is more thorough than mine were though.
I do remember that it never happened on my T430, but that's quite the
hardware gap.

Re: 6.9/amd64 runaway acpi process on Thinkpad T580

[David Anthony]

Another T480 user who has noticed the same problem. Per advice given, 
I've just enabled "BIOS Thunderbolt Assist". I will report back if I 
notice the problem persists.


On 9/19/21 4:50 AM, Daniel Wilkins wrote:

I've ran into this on my T480, it seems most consistently triggered by power
cycles caused by running out of battery. The bug's existed for quite a few
years (I think I first noticed it in 2019.) If I recall correctly I've
posted it to the list a couple of times but I don't think any concrete answers
ever emerged; your report is more thorough than mine were though.
I do remember that it never happened on my T430, but that's quite the
hardware gap.

Re: SOLVED Re: 6.9/amd64 runaway acpi process on Thinkpad T580

[Daniel Wilkins]

On Wed, Sep 29, 2021 at 11:47:34AM -0600, Theo de Raadt wrote:
> It would be great if someone figures out why "BIOS Thunderbolt Assist"
> disable, causes a pin to get stuck on resume, and/or figures out how we
> can recognize to handle/clear the event.

The detail in my BIOS options specifically mentions it as a Linux
workaround. Obviously patches couldn't be imported but I'll poke
around to see if there's any discussion/a description of what
exactly is happening.

Aside from that is there any data I can send y'all? Jonathan's built up
a pretty comprehensive set of dmesgs at this point, it seems like.

(No need to cc me, I'm on misc@)

Danny

SOLVED Re: 6.9/amd64 runaway acpi process on Thinkpad T580

[Jonathan Thornburg]

Hi,

On 2021-09-28 14>18>49, Daniel Wilkins wrote
> All you have to do is go into your bios' settings and turn on
> "BIOS Thunderbolt Assist" then everything will work 100% fine.
> 
> Thanks to jcs on IRC for pointing me at that (dunno what his
> email is.)

Success!  With this (and the 7.0 snapshot I installed yesterday; dmesg
in my message )
the problem is gone, and my T580 now does suspend/resume perfectly
(including idling with CPU usage under 1%).

A big thank-you to Daniel and to jcs (I'm guessing that's Joshua Stein,
https://jcs.org/) for the solution, and to Theo and Mike for their
suggestions too!

Thanks again,

--
-- "Jonathan Thornburg [remove color- to reply]" 
   on the west coast of Canada, eh?
   "There was of course no way of knowing whether you were being watched
at any given moment.  How often, or on what system, the Thought Police
plugged in on any individual wire was guesswork.  It was even conceivable
that they watched everybody all the time."  -- George Orwell, "1984"

Re: SOLVED Re: 6.9/amd64 runaway acpi process on Thinkpad T580

[Theo de Raadt]

Jonathan Thornburg  wrote:

> On 2021-09-28 14>18>49, Daniel Wilkins wrote
> > All you have to do is go into your bios' settings and turn on
> > "BIOS Thunderbolt Assist" then everything will work 100% fine.
> > 
> > Thanks to jcs on IRC for pointing me at that (dunno what his
> > email is.)
> 
> Success!  With this (and the 7.0 snapshot I installed yesterday; dmesg
> in my message )
> the problem is gone, and my T580 now does suspend/resume perfectly
> (including idling with CPU usage under 1%).
> 
> A big thank-you to Daniel and to jcs (I'm guessing that's Joshua Stein,
> https://jcs.org/) for the solution, and to Theo and Mike for their
> suggestions too!

It would be great if someone figures out why "BIOS Thunderbolt Assist"
disable, causes a pin to get stuck on resume, and/or figures out how we
can recognize to handle/clear the event.

Re: 6.9/amd64 runaway acpi process on Thinkpad T580

[Daniel Wilkins]

On Tue, Sep 28, 2021 at 10:08:47PM -0600, Theo de Raadt wrote:
> There are a few people who have experience with this.  Maybe one of
> them will mail you privately.
>

I'm glad this thread suddenly got revived, since I tried to find it
in my backlog but it got lost.

All you have to do is go into your bios' settings and turn on
"BIOS Thunderbolt Assist" then everything will work 100% fine.

Thanks to jcs on IRC for pointing me at that (dunno what his
email is.)

Re: 6.9/amd64 runaway acpi process on Thinkpad T580

[Jonathan Thornburg]

On Tue, Sep 28, 2021 at 10:16:23PM -0600, Theo de Raadt wrote:
> Your dmesg lacks tpm0.  You probably disabled it in the BIOS:
> 
> "STM7304" at acpi0 not configured
> 
> If you re-enable TPM uit in the BIOS, and try a snapshot (or upcoming
> 7.0) there is a recent fix which may help.  It is a potential reason for
> the interrupts...

I have re-enabled TPM in the BIOS.  Alas, a freshly installed snapshot
(dmesg below) still shows the same problem (1 core at 100% CPU usage
running what 'top -S -i -s1' says is 'acpi0') after doing a suspend/resume.

Tomorrow I will try Mike Larkin's suggestion of an ACPI_DEBUG kernel
and zzz/un-zzz from the text console.

--- begin snapshot dmesg ---
OpenBSD 7.0 (GENERIC.MP) #231: Mon Sep 27 17:23:17 MDT 2021
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 16755720192 (15979MB)
avail mem = 16231878656 (15479MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 3.0 @ 0xa86db000 (62 entries)
bios0: vendor LENOVO version "N27ET43W (1.29 )" date 08/13/2021
bios0: LENOVO 20L9001GUS
acpi0 at bios0: ACPI 5.0
acpi0: sleep states S0 S3 S4 S5
acpi0: tables DSDT FACP SSDT SSDT TPM2 UEFI SSDT SSDT HPET APIC MCFG ECDT SSDT 
SSDT BOOT BATB SLIC SSDT SSDT SSDT LPIT WSMT SSDT SSDT SSDT DBGP DBG2 MSDM DMAR 
ASF! FPDT UEFI
acpi0: wakeup devices GLAN(S4) XHC_(S3) XDCI(S4) HDAS(S4) RP01(S4) PXSX(S4) 
RP02(S4) PXSX(S4) PXSX(S4) RP04(S4) PXSX(S4) RP05(S4) PXSX(S4) RP06(S4) 
PXSX(S4) RP07(S4) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpihpet0 at acpi0: 2399 Hz
acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz, 1793.88 MHz, 06-8e-0a
cpu0: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SRBDS_CTRL,MD_CLEAR,TSXFA,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 24MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4.1.1.1, IBE
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz, 1794.44 MHz, 06-8e-0a
cpu1: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SRBDS_CTRL,MD_CLEAR,TSXFA,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 4 (application processor)
cpu2: Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz, 1795.82 MHz, 06-8e-0a
cpu2: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SRBDS_CTRL,MD_CLEAR,TSXFA,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu2: 256KB 64b/line 8-way L2 cache
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 6 (application processor)
cpu3: Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz, 1795.82 MHz, 06-8e-0a
cpu3: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SRBDS_CTRL,MD_CLEAR,TSXFA,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu3: 256KB 64b/line 8-way L2 cache
cpu3: smt 0, core 3, package 0
ioapic0 at mainbus0: apid 2 pa 0xfec0, version 20, 120 pins
acpimcfg0 at acpi0
acpimcfg0: addr 0xf000, bus 0-127
acpiec0 at acpi0
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 2 (RP01)
acpiprt2 at acpi0: bus -1 (RP02)
acpiprt3 at acpi0: bus -1 (RP03)
acpiprt4 at acpi0: bus -1 (RP04)

Re: 6.9/amd64 runaway acpi process on Thinkpad T580

[Mike Larkin]

On Tue, Sep 28, 2021 at 10:08:47PM -0600, Theo de Raadt wrote:
> the term "runaway ACPI" is not the best.  What is probably happening
> is a stuck interrupt.
>
> We continue to fight these.  Some of them are BIOS bugs, some are
> undocumented behaviours, sometimes AML parse errors in setting things
> up, and potentially a few are due to incorrect resume sequencing.
> The suspend/resume specification is weak, and getting even weaker as
> time goes by and newer machines come out which are poorly tested by
> even the mainstream OS vendors.
>
> Jonathan Thornburg  wrote:
>
> > After more experimentation, I find that the runaway ACPI process occurs
> > every time I suspend/resume (Fn-backspace).  (The system resumes fine
> > apart from the runaway ACPI process.)
> >
> > Is there any to kill or reset the kernel ACPI process short of rebooting?
> > /ps/ doen't see it, and /pkill/ (even /pkill -9/) has no effect.
>
> No you cannot kill kernel threads...
>
> > I will try compiling a custom kernel with ACPITHINKPAD_DEBUG defined
> > in /usr/src/sys/dev/acpi/acpithinkpad.c and see if that prints anything
> > interesting.  Are there any other particularly useful debugging things
> > I should explore to help track down the problem?
>
> There are a few people who have experience with this.  Maybe one of
> them will mail you privately.
>

If you build an ACPI_DEBUG kernel and zzz/un-zzz from the text console
(not X), you might see what GPE is stuck. it will probably be spewing tons
of debug output but maybe you can see which GPE it is.

-ml

Re: 6.9/amd64 runaway acpi process on Thinkpad T580

[Theo de Raadt]

> bios0: vendor LENOVO version "N27ET43W (1.29 )" date 08/13/2021
> bios0: LENOVO 20L9001GUS
> acpi0 at bios0: ACPI 5.0
> acpi0: sleep states S0 S3 S4 S5

On the other hand, your BIOS is very new.  So new that it has S0.
These days Microsoft is only testing S0.

Lenovo and some other vendors are re-adding S3, because S0 suspend is a
festering pile of crap which only works in Windows, well barely, on a
good day maybe.

But the S3 re-added is still very new BIOS (SMI?) emulation and it
has glitches.  It will take some time to mature.

You have a tremendous amount of wakeup devices which could be implicated
in this:

acpi0: wakeup devices GLAN(S4) XHC_(S3) XDCI(S4) HDAS(S4) RP01(S4) PXSX(S4) 
RP02(S4) PXSX(S4) PXSX(S4) RP04(S4) PXSX(S4) RP05(S4) PXSX(S4) RP06(S4) 
PXSX(S4) RP07(S4) [...]

Your dmesg lacks tpm0.  You probably disabled it in the BIOS:

"STM7304" at acpi0 not configured

If you re-enable TPM uit in the BIOS, and try a snapshot (or upcoming
7.0) there is a recent fix which may help.  It is a potential reason for
the interrupts...

Re: 6.9/amd64 runaway acpi process on Thinkpad T580

[Theo de Raadt]

BTW, BIOS update has fixed interrupts issues like this in a surprising
number of cases.  No promises, tho.

Jonathan Thornburg  wrote:

> After more experimentation, I find that the runaway ACPI process occurs
> every time I suspend/resume (Fn-backspace).  (The system resumes fine
> apart from the runaway ACPI process.)
> 
> Is there any to kill or reset the kernel ACPI process short of rebooting?
> /ps/ doen't see it, and /pkill/ (even /pkill -9/) has no effect.
> 
> I will try compiling a custom kernel with ACPITHINKPAD_DEBUG defined
> in /usr/src/sys/dev/acpi/acpithinkpad.c and see if that prints anything
> interesting.  Are there any other particularly useful debugging things
> I should explore to help track down the problem?
> 
> --
> -- "Jonathan Thornburg [remove color- to reply]" 
>on the west coast of Canada, eh?
>"There was of course no way of knowing whether you were being watched
> at any given moment.  How often, or on what system, the Thought Police
> plugged in on any individual wire was guesswork.  It was even conceivable
> that they watched everybody all the time."  -- George Orwell, "1984"
> 

Re: 6.9/amd64 runaway acpi process on Thinkpad T580

[Theo de Raadt]

the term "runaway ACPI" is not the best.  What is probably happening
is a stuck interrupt.  

We continue to fight these.  Some of them are BIOS bugs, some are
undocumented behaviours, sometimes AML parse errors in setting things
up, and potentially a few are due to incorrect resume sequencing.
The suspend/resume specification is weak, and getting even weaker as
time goes by and newer machines come out which are poorly tested by
even the mainstream OS vendors.

Jonathan Thornburg  wrote:

> After more experimentation, I find that the runaway ACPI process occurs
> every time I suspend/resume (Fn-backspace).  (The system resumes fine
> apart from the runaway ACPI process.)
> 
> Is there any to kill or reset the kernel ACPI process short of rebooting?
> /ps/ doen't see it, and /pkill/ (even /pkill -9/) has no effect.

No you cannot kill kernel threads...

> I will try compiling a custom kernel with ACPITHINKPAD_DEBUG defined
> in /usr/src/sys/dev/acpi/acpithinkpad.c and see if that prints anything
> interesting.  Are there any other particularly useful debugging things
> I should explore to help track down the problem?

There are a few people who have experience with this.  Maybe one of
them will mail you privately.

Re: 6.9/amd64 runaway acpi process on Thinkpad T580

[Jonathan Thornburg]

After more experimentation, I find that the runaway ACPI process occurs
every time I suspend/resume (Fn-backspace).  (The system resumes fine
apart from the runaway ACPI process.)

Is there any to kill or reset the kernel ACPI process short of rebooting?
/ps/ doen't see it, and /pkill/ (even /pkill -9/) has no effect.

I will try compiling a custom kernel with ACPITHINKPAD_DEBUG defined
in /usr/src/sys/dev/acpi/acpithinkpad.c and see if that prints anything
interesting.  Are there any other particularly useful debugging things
I should explore to help track down the problem?

--
-- "Jonathan Thornburg [remove color- to reply]" 
   on the west coast of Canada, eh?
   "There was of course no way of knowing whether you were being watched
at any given moment.  How often, or on what system, the Thought Police
plugged in on any individual wire was guesswork.  It was even conceivable
that they watched everybody all the time."  -- George Orwell, "1984"

Re: 6.9/amd64 runaway acpi process on Thinkpad T580

[Daniel Wilkins]

I dunno if this is helpful, but I just unplugged my thinkpad and triggered the 
behavior.

ACPI shot right up, and in this case the "charging" LED has stayed on. I've 
never triggered
it by unplugging before, but the symptoms are the same. The system was under 
some load while
doing so (watching a video in Firefox and extracting a backup.) The last line 
in dmesg also
seems weird to me; it might be a firmware thing, from that.

Danny
OpenBSD 7.0 (GENERIC.MP) #224: Mon Sep 20 11:44:33 MDT 2021
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 38362574848 (36585MB)
avail mem = 37183885312 (35461MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 3.0 @ 0x6ecc4000 (63 entries)
bios0: vendor LENOVO version "N24ET49W (1.24 )" date 04/19/2019
bios0: LENOVO 20L50054US
acpi0 at bios0: ACPI 5.0
acpi0: sleep states S0 S3 S4 S5
acpi0: tables DSDT FACP SSDT SSDT TPM2 UEFI SSDT SSDT HPET APIC MCFG ECDT SSDT 
SSDT SSDT BOOT BATB SLIC SSDT SSDT SSDT LPIT WSMT SSDT SSDT SSDT DBGP DBG2 MSDM 
DMAR ASF! FPDT UEFI
acpi0: wakeup devices GLAN(S4) XHC_(S3) XDCI(S4) HDAS(S4) RP01(S4) PXSX(S4) 
RP02(S4) PXSX(S4) RP03(S4) PXSX(S4) RP04(S4) PXSX(S4) RP05(S4) PXSX(S4) 
RP06(S4) PXSX(S4) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpihpet0 at acpi0: 2399 Hz
acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Core(TM) i5-8350U CPU @ 1.70GHz, 1591.45 MHz, 06-8e-0a
cpu0: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SRBDS_CTRL,MD_CLEAR,TSXFA,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 24MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4.1.1.1, IBE
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Core(TM) i5-8350U CPU @ 1.70GHz, 1596.28 MHz, 06-8e-0a
cpu1: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SRBDS_CTRL,MD_CLEAR,TSXFA,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 4 (application processor)
cpu2: Intel(R) Core(TM) i5-8350U CPU @ 1.70GHz, 1596.28 MHz, 06-8e-0a
cpu2: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SRBDS_CTRL,MD_CLEAR,TSXFA,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu2: 256KB 64b/line 8-way L2 cache
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 6 (application processor)
cpu3: Intel(R) Core(TM) i5-8350U CPU @ 1.70GHz, 1596.28 MHz, 06-8e-0a
cpu3: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SRBDS_CTRL,MD_CLEAR,TSXFA,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu3: 256KB 64b/line 8-way L2 cache
cpu3: smt 0, core 3, package 0
cpu4 at mainbus0: apid 1 (application processor)
cpu4: Intel(R) Core(TM) i5-8350U CPU @ 1.70GHz, 1596.28 MHz, 06-8e-0a
cpu4: 

Re: 6.9/amd64 runaway acpi process on Thinkpad T580

[Jonathan Thornburg]

I wrote

> During the installation (both in the bsd.rd install script and previously
> when I dropped into the bsd.rd shell to set up softraid-crypto) the machine
> acted incredibly slow, and there was a several-second delay in echoing
> typed characters.  I suspected that it was some device producing spurious
> interrupts, and just let the install run overnight until it finally
> finished.

I neglected to note that I also saw similar behavior with another T580
I previously bought (& then returned when it proved to have hardware
defects).  Combined with Daniel Wilkins' experience with a T480, this
suggests that this is a generic problem with Thinkpad T[45]80.  Does
anyone have a T[45]80 who has *not* seen this problem?

--
-- "Jonathan Thornburg [remove color- to reply]" 
   on the west coast of Canada, eh?
   "There was of course no way of knowing whether you were being watched
at any given moment.  How often, or on what system, the Thought Police
plugged in on any individual wire was guesswork.  It was even conceivable
that they watched everybody all the time."  -- George Orwell, "1984"

Re: 6.9/amd64 runaway acpi process on Thinkpad T580

[Daniel Wilkins]

I've ran into this on my T480, it seems most consistently triggered by power
cycles caused by running out of battery. The bug's existed for quite a few
years (I think I first noticed it in 2019.) If I recall correctly I've
posted it to the list a couple of times but I don't think any concrete answers
ever emerged; your report is more thorough than mine were though.
I do remember that it never happened on my T430, but that's quite the
hardware gap.

6.9/amd64 runaway acpi process on Thinkpad T580

[Jonathan Thornburg]

I have just installed 6.9-stable/amd64 on a new-to-me (used) Lenovo
Thinkpad T580 (dmesg below).  This was a from-scratch install on a
new-from-the-factory SSD (via booting the 6.9/amd64 bsd.rd from a usb
stick).

During the installation (both in the bsd.rd install script and previously
when I dropped into the bsd.rd shell to set up softraid-crypto) the machine
acted incredibly slow, and there was a several-second delay in echoing
typed characters.  I suspected that it was some device producing spurious
interrupts, and just let the install run overnight until it finally
finished.

After the install (booting into normal multiuser operation) the machine
seemed to work fine at first.  Notably, X "just works", screen brightness
adjust with Fn-F5/Fn-F6 "just works", iwm wifi "just works", and
suspend-to-RAM with Fn/Backspace "just works".

*BUT*, intermittently (maybe 25% of the time?) after a power-cycle and
reboot, there is what appears to be a system process 'acpi0' infinite-looping
(taking 100% of one CPU core, with 'top' showing ~80% system time for that
processor).  Here's a cut-n-paste of the beginning of 'top -S -i -s1'
output in that state, showing the runaway process:

load averages:  1.02,  1.10,  0.86   gold.bkis-orchard.net 00:41:44
134 processes: 130 idle, 4 on processorup  0:19
CPU0:  0.0% user,  0.0% nice, 80.2% sys,  1.0% spin, 17.8% intr,  1.0% idle
CPU1:  0.0% user,  0.0% nice,  0.0% sys,  1.0% spin,  0.0% intr, 99.0% idle
CPU2:  0.0% user,  0.0% nice,  1.0% sys,  2.0% spin,  0.0% intr, 97.0% idle
CPU3:  0.0% user,  0.0% nice,  1.0% sys,  2.0% spin,  0.0% intr, 97.0% idle
Memory: Real: 341M/1548M act/tot Free: 14G Cache: 665M Swap: 0K/34G

  PID USERNAME PRI NICE  SIZE   RES STATE WAIT  TIMECPU COMMAND
67563 root  1000K   19M sleep/0   acpi014:48 77.73% acpi0
59020 _x11   20   36M   59M sleep/1   poll  0:48  5.47% Xorg
48374 root   20 8952K   15M sleep/2   select0:09  0.73% perl

Specifying an additional '-H' option to 'top' ("show process threads")
didn't change the output significantly.

FWIW, I have apmd running

# cat /etc/rc.conf.local
apmd_flags='-A -t 60'
vmd_flags=''
xenodm_flags=''
#

apmd appears to be adjusting the CPU clock rate correctly, both now
and on those cold-boots where the infinite-loop problem does not occur.
As I noted above, suspend-to-RAM (via Fn-Backspace, which is the key
combination marked with the usual Thinkpad "moon" icon) "just works".

Is this sort of acpi (?) runaway a known T580 problem?  Neither google
nor the nycbug.org dmesg archive show any OpenBSD T580 dmesg, but I do
see occasional web posts mentioning OpenBSD on a T580.

Below I give my dmesg (from the current boot, the one that produced the
above runaway process).  What other information would be useful to try
to diagnose the problem?

Thanks,
--
-- "Jonathan Thornburg [remove color- to reply]" 
   on the west coast of Canada, eh?

--- begin dmesg ---
OpenBSD 6.9 (GENERIC.MP) #4: Tue Aug 10 08:12:23 MDT 2021

r...@syspatch-69-amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 16755720192 (15979MB)
avail mem = 16232525824 (15480MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 3.0 @ 0xa86db000 (62 entries)
bios0: vendor LENOVO version "N27ET43W (1.29 )" date 08/13/2021
bios0: LENOVO 20L9001GUS
acpi0 at bios0: ACPI 5.0
acpi0: sleep states S0 S3 S4 S5
acpi0: tables DSDT FACP SSDT SSDT TPM2 UEFI SSDT SSDT HPET APIC MCFG ECDT SSDT 
SSDT BOOT BATB SLIC SSDT SSDT SSDT LPIT WSMT SSDT SSDT SSDT DBGP DBG2 MSDM DMAR 
ASF! FPDT UEFI
acpi0: wakeup devices GLAN(S4) XHC_(S3) XDCI(S4) HDAS(S4) RP01(S4) PXSX(S4) 
RP02(S4) PXSX(S4) PXSX(S4) RP04(S4) PXSX(S4) RP05(S4) PXSX(S4) RP06(S4) 
PXSX(S4) RP07(S4) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpihpet0 at acpi0: 2399 Hz
acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz, 1794.33 MHz, 06-8e-0a
cpu0: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SRBDS_CTRL,MD_CLEAR,TSXFA,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 24MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4.1.1.1, IBE
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz, 1794.19 MHz, 06-8e-0a
cpu1: