Re: Status of Intel Hybrid CPU support (Alder Lake/Raptor Lake) support

[Konstantin Belousov]

On Tue, Jan 03, 2023 at 08:20:16AM +, Chen, Alvin W wrote:
> > 
> > [EXTERNAL EMAIL]
> > 
> > On 2022-11-14 09:09 +0200, Konstantin Belousov wrote:
> > >
> > > You might use this patch meantime
> > > https://urldefense.com/v3/__https://kib.kiev.ua/git/gitweb.cgi?p=devia
> > >
> > nt3.git;a=commit;h=5d72240a8777b26d5e0a7d2d26bb919d05f60002__;!!Lp
> > KI!j
> > >
> > pyHChyB8NZAQq5isiNFepD61cX0HczrFCeOriYbSwyfMPGW7k8I_BYxPWXv1FG
> > yfl1Y-Ip
> > > dgcNkyg$ [kib[.]kiev[.]ua]
> > 
> > Thank you for this patch.  On my 13.1 machine it fixed all the panics with 
> > a i9-
> > 12900KF.  It's nice to be able to use all the cores since I got this 
> > machine in
> > May.  I had them disabled in BIOS.
> > 
> > Also as a side note this fixed the audio issues I was having with the e 
> > cores
> > both enabled and disabled.
> > 
> 
> This patch can't work well with some older CPUs, and I find one N6005 (Intel 
> JSP).
> This JSP cpu is not a hybrid cpu architecture, but the high cpu id is 0x1b 
> with 0x2000 which meet the small core condition in the patch, but kernel 
> got crashed on this cpu.
> We need a better condition to pick up the E cores to make sure the kernel can 
> be compatible with all older Non-Hybrid CPUs.

How does the kernel panic?  Could you show the verbose dmesg lines with
CPUID information?

If this is JasperLake, I suspect that the patch below should be enough.

Still, the workaround is not confirmed/explained by you (Intel).  I have
a high hope that we would eventually be able to use INVLPG with PCID on
(newer) small cores, might be even on older small cores after a microcode
update.

commit c8812792202d73804cccd624f6478c4069b22438
Author: Konstantin Belousov 
Date:   Tue Jan 3 12:13:07 2023 +0200

amd64: be more precise when enabling the AlderLake workaround

Reported by:"Chen, Alvin W" 
Sponsored by:   The FreeBSD Foundation
MFC after:  1 week

diff --git a/sys/amd64/amd64/initcpu.c b/sys/amd64/amd64/initcpu.c
index 08385d3095d0..cddf8502437e 100644
--- a/sys/amd64/amd64/initcpu.c
+++ b/sys/amd64/amd64/initcpu.c
@@ -247,6 +247,26 @@ cpu_auxmsr(void)
return (PCPU_GET(cpuid));
 }
 
+void
+cpu_init_small_core(void)
+{
+   u_int r[4];
+
+   if (cpu_high < 0x1a)
+   return;
+
+   cpuid_count(0x1a, 0, r);
+   if ((r[0] & CPUID_HYBRID_CORE_MASK) != CPUID_HYBRID_SMALL_CORE)
+   return;
+
+   PCPU_SET(small_core, 1);
+   if (pmap_pcid_enabled && invpcid_works &&
+   pmap_pcid_invlpg_workaround_uena) {
+   PCPU_SET(pcid_invlpg_workaround, 1);
+   pmap_pcid_invlpg_workaround = 1;
+   }
+}
+
 /*
  * Initialize CPU control registers
  */
@@ -255,7 +275,6 @@ initializecpu(void)
 {
uint64_t msr;
uint32_t cr4;
-   u_int r[4];
 
cr4 = rcr4();
if ((cpu_feature & CPUID_XMM) && (cpu_feature & CPUID_FXSR)) {
@@ -319,18 +338,8 @@ initializecpu(void)
(cpu_stdext_feature2 & CPUID_STDEXT2_RDPID) != 0)
wrmsr(MSR_TSC_AUX, cpu_auxmsr());
 
-   if (cpu_high >= 0x1a) {
-   cpuid_count(0x1a, 0, r);
-   if ((r[0] & CPUID_HYBRID_CORE_MASK) ==
-   CPUID_HYBRID_SMALL_CORE) {
-   PCPU_SET(small_core, 1);
-   if (pmap_pcid_enabled &&
-   pmap_pcid_invlpg_workaround_uena) {
-   PCPU_SET(pcid_invlpg_workaround, 1);
-   pmap_pcid_invlpg_workaround = 1;
-   }
-   }
-   }
+   if (!IS_BSP())
+   cpu_init_small_core();
 }
 
 void
diff --git a/sys/amd64/amd64/machdep.c b/sys/amd64/amd64/machdep.c
index 05342b31d2aa..c601ce868978 100644
--- a/sys/amd64/amd64/machdep.c
+++ b/sys/amd64/amd64/machdep.c
@@ -1341,6 +1341,12 @@ hammer_time(u_int64_t modulep, u_int64_t physfree)
pmap_pcid_enabled = 0;
}
 
+   /*
+* Now we can do small core initialization, after the PCID
+* CPU features and user knobs are evaluated.
+*/
+   cpu_init_small_core();
+
link_elf_ireloc(kmdp);
 
/*
diff --git a/sys/amd64/include/md_var.h b/sys/amd64/include/md_var.h
index f014c66c0d06..f5cbdb6bbd9d 100644
--- a/sys/amd64/include/md_var.h
+++ b/sys/amd64/include/md_var.h
@@ -72,6 +72,7 @@ void  amd64_bsp_ist_init(struct pcpu *pc);
 void   amd64_syscall(struct thread *td, int traced);
 void   amd64_syscall_ret_flush_l1d(int error);
 void   amd64_syscall_ret_flush_l1d_recalc(void);
+void   cpu_init_small_core(void);
 void   doreti_iret(void) __asm(__STRING(doreti_iret));
 void   doreti_iret_fault(void) __asm(__STRING(doreti_iret_fault));
 void   flush_l1d_sw_abi(void);

RE: Status of Intel Hybrid CPU support (Alder Lake/Raptor Lake) support

[Chen, Alvin W]

> 
> [EXTERNAL EMAIL]
> 
> On 2022-11-14 09:09 +0200, Konstantin Belousov wrote:
> >
> > You might use this patch meantime
> > https://urldefense.com/v3/__https://kib.kiev.ua/git/gitweb.cgi?p=devia
> >
> nt3.git;a=commit;h=5d72240a8777b26d5e0a7d2d26bb919d05f60002__;!!Lp
> KI!j
> >
> pyHChyB8NZAQq5isiNFepD61cX0HczrFCeOriYbSwyfMPGW7k8I_BYxPWXv1FG
> yfl1Y-Ip
> > dgcNkyg$ [kib[.]kiev[.]ua]
> 
> Thank you for this patch.  On my 13.1 machine it fixed all the panics with a 
> i9-
> 12900KF.  It's nice to be able to use all the cores since I got this machine 
> in
> May.  I had them disabled in BIOS.
> 
> Also as a side note this fixed the audio issues I was having with the e cores
> both enabled and disabled.
> 

This patch can't work well with some older CPUs, and I find one N6005 (Intel 
JSP).
This JSP cpu is not a hybrid cpu architecture, but the high cpu id is 0x1b with 
0x2000 which meet the small core condition in the patch, but kernel got 
crashed on this cpu.
We need a better condition to pick up the E cores to make sure the kernel can 
be compatible with all older Non-Hybrid CPUs.

Internal Use - Confidential

Re: Status of Intel Hybrid CPU support (Alder Lake/Raptor Lake) support

[Amar Takhar]

On 2022-11-14 09:09 +0200, Konstantin Belousov wrote:
>
> You might use this patch meantime
> https://kib.kiev.ua/git/gitweb.cgi?p=deviant3.git;a=commit;h=5d72240a8777b26d5e0a7d2d26bb919d05f60002

Thank you for this patch.  On my 13.1 machine it fixed all the panics with a
i9-12900KF.  It's nice to be able to use all the cores since I got this machine 
in May.  I had them disabled in BIOS.

Also as a side note this fixed the audio issues I was having with the e cores 
both enabled and disabled.

  https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=263385


Thanks again!


Amar.

RE: Status of Intel Hybrid CPU support (Alder Lake/Raptor Lake) support

[Chen, Alvin W]

> > I am not familiar with TLB part implementation for BSD. Based on your
> patch, it looks like disable PCID or force flush TLB for E core. Is that 
> right?
> > Would you mind explain a little more to help understand the code?
> 
> The patch does what its title said. On small cores it does not rely on INVLPG
> to flush global TLB entries.  Instead, total flush of TLB with INVPCID
> instruction is performed.
> 
> For large cores, no change in behavior is intended.

Thanks. I apply the patch and test for 10 hours+ on my ADL-P laptop. It works 
well, and the issue is never reproduced.

Internal Use - Confidential

Re: Status of Intel Hybrid CPU support (Alder Lake/Raptor Lake) support

[Konstantin Belousov]

On Tue, Nov 15, 2022 at 07:49:55AM +, Chen, Alvin W wrote:
> Kongstantin,
> 
> I am not familiar with TLB part implementation for BSD. Based on your patch, 
> it looks like disable PCID or force flush TLB for E core. Is that right?
> Would you mind explain a little more to help understand the code?

The patch does what its title said. On small cores it does not rely on
INVLPG to flush global TLB entries.  Instead, total flush of TLB with
INVPCID instruction is performed.

For large cores, no change in behavior is intended.
> 
> 
> Internal Use - Confidential
> 
> > -Original Message-
> > From: owner-freebsd-curr...@freebsd.org  > curr...@freebsd.org> On Behalf Of Amar Takhar
> > Sent: Tuesday, November 15, 2022 1:48 AM
> > To: freebsd-current@freebsd.org
> > Subject: Re: Status of Intel Hybrid CPU support (Alder Lake/Raptor Lake)
> > support
> > 
> > 
> > [EXTERNAL EMAIL]
> > 
> > On 2022-11-14 09:09 +0200, Konstantin Belousov wrote:
> > >
> > > You might use this patch meantime
> > > https://urldefense.com/v3/__https://kib.kiev.ua/git/gitweb.cgi?p=devia
> > >
> > nt3.git;a=commit;h=5d72240a8777b26d5e0a7d2d26bb919d05f60002__;!!LpKI
> > !g
> > > -6xRexMgrS7blkbBAEW-CU6xx2soVgqJcn34v5a-
> > vYybodWNslFhcgFr631abOyXATJf_r
> > > ACTtzog$ [kib[.]kiev[.]ua]
> > 
> > I know this is -CURRENT but will this work on 13.1 as well?  I use that as 
> > my
> > main workstation.
> > 
> > Also does vm.pmap.pcid_enabled need to be set to 0 still?
> > 
> > Hopefully this fixes the sound issues as well right now I have all my 
> > E-cores
> > disabled but sound issues persist.
> > 
> > Thanks!
> > 
> > 
> > Amar.

RE: Status of Intel Hybrid CPU support (Alder Lake/Raptor Lake) support

[Chen, Alvin W]

Kongstantin,

I am not familiar with TLB part implementation for BSD. Based on your patch, it 
looks like disable PCID or force flush TLB for E core. Is that right?
Would you mind explain a little more to help understand the code?


Internal Use - Confidential

> -Original Message-
> From: owner-freebsd-curr...@freebsd.org  curr...@freebsd.org> On Behalf Of Amar Takhar
> Sent: Tuesday, November 15, 2022 1:48 AM
> To: freebsd-current@freebsd.org
> Subject: Re: Status of Intel Hybrid CPU support (Alder Lake/Raptor Lake)
> support
> 
> 
> [EXTERNAL EMAIL]
> 
> On 2022-11-14 09:09 +0200, Konstantin Belousov wrote:
> >
> > You might use this patch meantime
> > https://urldefense.com/v3/__https://kib.kiev.ua/git/gitweb.cgi?p=devia
> >
> nt3.git;a=commit;h=5d72240a8777b26d5e0a7d2d26bb919d05f60002__;!!LpKI
> !g
> > -6xRexMgrS7blkbBAEW-CU6xx2soVgqJcn34v5a-
> vYybodWNslFhcgFr631abOyXATJf_r
> > ACTtzog$ [kib[.]kiev[.]ua]
> 
> I know this is -CURRENT but will this work on 13.1 as well?  I use that as my
> main workstation.
> 
> Also does vm.pmap.pcid_enabled need to be set to 0 still?
> 
> Hopefully this fixes the sound issues as well right now I have all my E-cores
> disabled but sound issues persist.
> 
> Thanks!
> 
> 
> Amar.

Re: Status of Intel Hybrid CPU support (Alder Lake/Raptor Lake) support

[Amar Takhar]

On 2022-11-14 09:09 +0200, Konstantin Belousov wrote:
> 
> You might use this patch meantime
> https://kib.kiev.ua/git/gitweb.cgi?p=deviant3.git;a=commit;h=5d72240a8777b26d5e0a7d2d26bb919d05f60002

I know this is -CURRENT but will this work on 13.1 as well?  I use that as my 
main workstation.

Also does vm.pmap.pcid_enabled need to be set to 0 still?

Hopefully this fixes the sound issues as well right now I have all my E-cores 
disabled but sound issues persist.

Thanks!


Amar.

Re: Status of Intel Hybrid CPU support (Alder Lake/Raptor Lake) support

[Konstantin Belousov]

On Mon, Nov 14, 2022 at 06:50:20AM +, Chen, Alvin W wrote:
> 
> 
> Internal Use - Confidential
> 
> 
> > -Original Message-
> > From: Mike Karels 
> > Sent: 2022年10月27日 23:17
> > To: Chen, Alvin W
> > Cc: freebsd-current
> > Subject: Re: Status of Intel Hybrid CPU support (Alder Lake/Raptor Lake)
> > support
> > 
> > 
> > [EXTERNAL EMAIL]
> > 
> > On 26 Oct 2022, at 22:29, Chen, Alvin W wrote:
> > 
> > > Any body know the status to support P Core and E Core?
> > > Now WA is: disable PCID.
> > 
> > Disabling vm.pmap.pcid_enabled is the best workaround for now.
> > There has been some progress in investigations in the background, but I 
> > don’t
> > think there is a permanent solution yet.
> > 
> > btw, I haven’t heard reports about Raptor Lake yet.  Do you know if it has 
> > the
> > same problems as Alder Lake on hybrids?  I would guess it does.
> > 
> 
> I get one RPL-S DVT, and it hasn't mode name yet as it is the engineering CPU 
> intel provided for validation.
> Based on my testing, the issue is still there:
> 
> "Boot to shell by USB disk installer, and mount a FAT32 partition (on SSD), 
> and copy a 300MB file to the FAT32, compare the sha256 checksums for the 
> source file and the dst file, the checksum are different."

You might use this patch meantime
https://kib.kiev.ua/git/gitweb.cgi?p=deviant3.git;a=commit;h=5d72240a8777b26d5e0a7d2d26bb919d05f60002

RE: Status of Intel Hybrid CPU support (Alder Lake/Raptor Lake) support

[Chen, Alvin W]

Internal Use - Confidential


> -Original Message-
> From: Mike Karels 
> Sent: 2022年10月27日 23:17
> To: Chen, Alvin W
> Cc: freebsd-current
> Subject: Re: Status of Intel Hybrid CPU support (Alder Lake/Raptor Lake)
> support
> 
> 
> [EXTERNAL EMAIL]
> 
> On 26 Oct 2022, at 22:29, Chen, Alvin W wrote:
> 
> > Any body know the status to support P Core and E Core?
> > Now WA is: disable PCID.
> 
> Disabling vm.pmap.pcid_enabled is the best workaround for now.
> There has been some progress in investigations in the background, but I don’t
> think there is a permanent solution yet.
> 
> btw, I haven’t heard reports about Raptor Lake yet.  Do you know if it has the
> same problems as Alder Lake on hybrids?  I would guess it does.
> 

I get one RPL-S DVT, and it hasn't mode name yet as it is the engineering CPU 
intel provided for validation.
Based on my testing, the issue is still there:

"Boot to shell by USB disk installer, and mount a FAT32 partition (on SSD), and 
copy a 300MB file to the FAT32, compare the sha256 checksums for the source 
file and the dst file, the checksum are different."

Re: Status of Intel Hybrid CPU support (Alder Lake/Raptor Lake) support

[Mike Karels]

On 26 Oct 2022, at 22:29, Chen, Alvin W wrote:

> Any body know the status to support P Core and E Core?
> Now WA is: disable PCID.

Disabling vm.pmap.pcid_enabled is the best workaround for now.
There has been some progress in investigations in the background, but I
don’t think there is a permanent solution yet.

btw, I haven’t heard reports about Raptor Lake yet.  Do you know if it has
the same problems as Alder Lake on hybrids?  I would guess it does.

Mike
>
> Regards,
> Alvin Chen
> Dell ThinOS | Dell - Comercial Client Group
> Teams/Zoom: weike_c...@dell.com
>
>
>
> Internal Use - Confidential