Re: [v3] amd64: simplify TSC sync testing
Hi, >> On Jul 20, 2022, at 01:48, Masato Asou wrote: >> >> Sorry, my latest reply. >> >> I tested your patch on my Proxmox Virtual Environment on Ryzen7 box. >> It works fine for me. > > This VM doesn't have the ITSC CPU flag, > how is it using the TSC as a timecounter? > >> OpenBSD 7.1-current (GENERIC.MP) #1: Wed Jul 20 14:15:23 JST 2022 >>a...@pve-obsd.my.domain:/usr/src/sys/arch/amd64/compile/GENERIC.MP >> real mem = 17162952704 (16367MB) >> avail mem = 16625430528 (15855MB) >> random: good seed from bootblocks >> mpath0 at root >> scsibus0 at mpath0: 256 targets >> mainbus0 at root >> bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf59c0 (10 entries) >> bios0: vendor SeaBIOS version "rel-1.15.0-0-g2dd4b9b3f840-prebuilt.qemu.org" >> date 04/01/2014 >> bios0: QEMU Standard PC (i440FX + PIIX, 1996) >> acpi0 at bios0: ACPI 1.0 >> acpi0: sleep states S3 S4 S5 >> acpi0: tables DSDT FACP APIC SSDT HPET WAET >> acpi0: wakeup devices >> acpitimer0 at acpi0: 3579545 Hz, 24 bits >> acpimadt0 at acpi0 addr 0xfee0: PC-AT compat >> cpu0 at mainbus0: apid 0 (boot processor) >> cpu0: Common KVM processor, 3593.56 MHz, 0f-06-01 >> cpu0: >> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,CX16,x2APIC,HV,NXE,LONG,LAHF,CMPLEG > > Here, no "ITSC". Sorry, This VM did not use ITSC as you mentioned. $ sysctl kern.timecounter kern.timecounter.tick=1 kern.timecounter.timestepwarnings=0 kern.timecounter.hardware=pvclock0 kern.timecounter.choice=i8254(0) pvclock0(1500) acpihpet0(1000) acpitimer0(1000) -- ASOU Masato
Re: [v3] amd64: simplify TSC sync testing
> On Jul 20, 2022, at 01:48, Masato Asou wrote: > > Sorry, my latest reply. > > I tested your patch on my Proxmox Virtual Environment on Ryzen7 box. > It works fine for me. This VM doesn't have the ITSC CPU flag, how is it using the TSC as a timecounter? > OpenBSD 7.1-current (GENERIC.MP) #1: Wed Jul 20 14:15:23 JST 2022 >a...@pve-obsd.my.domain:/usr/src/sys/arch/amd64/compile/GENERIC.MP > real mem = 17162952704 (16367MB) > avail mem = 16625430528 (15855MB) > random: good seed from bootblocks > mpath0 at root > scsibus0 at mpath0: 256 targets > mainbus0 at root > bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf59c0 (10 entries) > bios0: vendor SeaBIOS version "rel-1.15.0-0-g2dd4b9b3f840-prebuilt.qemu.org" > date 04/01/2014 > bios0: QEMU Standard PC (i440FX + PIIX, 1996) > acpi0 at bios0: ACPI 1.0 > acpi0: sleep states S3 S4 S5 > acpi0: tables DSDT FACP APIC SSDT HPET WAET > acpi0: wakeup devices > acpitimer0 at acpi0: 3579545 Hz, 24 bits > acpimadt0 at acpi0 addr 0xfee0: PC-AT compat > cpu0 at mainbus0: apid 0 (boot processor) > cpu0: Common KVM processor, 3593.56 MHz, 0f-06-01 > cpu0: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,CX16,x2APIC,HV,NXE,LONG,LAHF,CMPLEG Here, no "ITSC".
Re: [v3] amd64: simplify TSC sync testing
Sorry, my latest reply. I tested your patch on my Proxmox Virtual Environment on Ryzen7 box. It works fine for me. OpenBSD 7.1-current (GENERIC.MP) #1: Wed Jul 20 14:15:23 JST 2022 a...@pve-obsd.my.domain:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 17162952704 (16367MB) avail mem = 16625430528 (15855MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf59c0 (10 entries) bios0: vendor SeaBIOS version "rel-1.15.0-0-g2dd4b9b3f840-prebuilt.qemu.org" date 04/01/2014 bios0: QEMU Standard PC (i440FX + PIIX, 1996) acpi0 at bios0: ACPI 1.0 acpi0: sleep states S3 S4 S5 acpi0: tables DSDT FACP APIC SSDT HPET WAET acpi0: wakeup devices acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Common KVM processor, 3593.56 MHz, 0f-06-01 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,CX16,x2APIC,HV,NXE,LONG,LAHF,CMPLEG cpu0: 64KB 64b/line 2-way D-cache, 64KB 64b/line 2-way I-cache cpu0: 512KB 64b/line 16-way L2 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges cpu0: apic clock running at 999MHz cpu1 at mainbus0: apid 1 (application processor) cpu1: Common KVM processor, 3593.23 MHz, 0f-06-01 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,CX16,x2APIC,HV,NXE,LONG,LAHF,CMPLEG cpu1: 64KB 64b/line 2-way D-cache, 64KB 64b/line 2-way I-cache cpu1: 512KB 64b/line 16-way L2 cache cpu1: smt 0, core 1, package 0 cpu2 at mainbus0: apid 2 (application processor) cpu2: Common KVM processor, 3593.22 MHz, 0f-06-01 cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,CX16,x2APIC,HV,NXE,LONG,LAHF,CMPLEG cpu2: 64KB 64b/line 2-way D-cache, 64KB 64b/line 2-way I-cache cpu2: 512KB 64b/line 16-way L2 cache cpu2: smt 0, core 2, package 0 cpu3 at mainbus0: apid 3 (application processor) cpu3: Common KVM processor, 3593.21 MHz, 0f-06-01 cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,CX16,x2APIC,HV,NXE,LONG,LAHF,CMPLEG cpu3: 64KB 64b/line 2-way D-cache, 64KB 64b/line 2-way I-cache cpu3: 512KB 64b/line 16-way L2 cache cpu3: smt 0, core 3, package 0 cpu4 at mainbus0: apid 4 (application processor) cpu4: Common KVM processor, 3593.23 MHz, 0f-06-01 cpu4: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,CX16,x2APIC,HV,NXE,LONG,LAHF,CMPLEG cpu4: 64KB 64b/line 2-way D-cache, 64KB 64b/line 2-way I-cache cpu4: 512KB 64b/line 16-way L2 cache cpu4: smt 0, core 4, package 0 cpu5 at mainbus0: apid 5 (application processor) cpu5: Common KVM processor, 3593.23 MHz, 0f-06-01 cpu5: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,CX16,x2APIC,HV,NXE,LONG,LAHF,CMPLEG cpu5: 64KB 64b/line 2-way D-cache, 64KB 64b/line 2-way I-cache cpu5: 512KB 64b/line 16-way L2 cache cpu5: smt 0, core 5, package 0 cpu6 at mainbus0: apid 6 (application processor) cpu6: Common KVM processor, 3593.22 MHz, 0f-06-01 cpu6: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,CX16,x2APIC,HV,NXE,LONG,LAHF,CMPLEG cpu6: 64KB 64b/line 2-way D-cache, 64KB 64b/line 2-way I-cache cpu6: 512KB 64b/line 16-way L2 cache cpu6: smt 0, core 6, package 0 cpu7 at mainbus0: apid 7 (application processor) cpu7: Common KVM processor, 3593.23 MHz, 0f-06-01 cpu7: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,CX16,x2APIC,HV,NXE,LONG,LAHF,CMPLEG cpu7: 64KB 64b/line 2-way D-cache, 64KB 64b/line 2-way I-cache cpu7: 512KB 64b/line 16-way L2 cache cpu7: smt 0, core 7, package 0 ioapic0 at mainbus0: apid 0 pa 0xfec0, version 11, 24 pins acpihpet0 at acpi0: 1 Hz acpiprt0 at acpi0: bus 0 (PCI0) "ACPI0006" at acpi0 not configured acpipci0 at acpi0 PCI0 acpicmos0 at acpi0 "PNP0A06" at acpi0 not configured "PNP0A06" at acpi0 not configured "PNP0A06" at acpi0 not configured "QEMU0002" at acpi0 not configured "ACPI0010" at acpi0 not configured "QEMUVGID" at acpi0 not configured acpicpu0 at acpi0: C1(@1 halt!) acpicpu1 at acpi0: C1(@1 halt!) acpicpu2 at acpi0: C1(@1 halt!) acpicpu3 at acpi0: C1(@1 halt!) acpicpu4 at acpi0: C1(@1 halt!) acpicpu5 at acpi0: C1(@1 halt!) acpicpu6 at acpi0: C1(@1 halt!) acpicpu7 at acpi0: C1(@1 halt!) pvbus0 at mainbus0: KVM pvclock0 at pvbus0 pci0 at mainbus0 bus 0 pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02 pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00 pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility wd0 at pciide0 channel 0 drive 0: wd0: 16-sector PIO, LBA48,
Re: [v3] amd64: simplify TSC sync testing
Christian Weisgerber wrote: > Scott Cheloha: > > > > kern.timecounter.tick=1 > > > kern.timecounter.timestepwarnings=0 > > > kern.timecounter.hardware=i8254 > > > kern.timecounter.choice=i8254(0) tsc(-1000) acpihpet0(1000) > > > acpitimer0(1000) > > > > This is expected behavior with the patch. > > > > cpu0's TSC is way out of sync with every > > other CPU's TSC, so the TSC is marked > > as a bad timecounter and a different one is > > chosen. > > Shouldn't it pick acpihpet0 then? Let's keep in mind the goal: When it works well, the TSC is a better timer than all the others, so if the TSC problems can be diagnosed & resolved by various measurements and mitigations, then we will want the TSC to be chosen. So the long term roadmap probably does NOT include acpihpet0 being chosen. Long term, that hopefully is a falacy.
Re: [v3] amd64: simplify TSC sync testing
Hi, I tested your patch on my OpenSUSE + Xen on Ryzen7 box. It works fine for me. OpenBSD 7.1-current (GENERIC.MP) #1: Thu Jul 7 15:22:32 JST 2022 a...@xen-obsd1.my.domain:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 8556376064 (8159MB) avail mem = 8279695360 (7896MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xfc10 (18 entries) bios0: vendor Xen version "4.14.5_02-150300.3." date 06/08/2022 bios0: Xen HVM domU acpi0 at bios0: ACPI 4.0 acpi0: sleep states S5 acpi0: tables DSDT FACP APIC HPET WAET SSDT SSDT acpi0: wakeup devices acpitimer0 at acpi0: 3579545 Hz, 32 bits acpimadt0 at acpi0 addr 0xfee0: PC-AT compat ioapic0 at mainbus0: apid 1 pa 0xfec0, version 11, 48 pins, remapped cpu0 at mainbus0: apid 0 (boot processor) cpu0: AMD Ryzen 7 3700X 8-Core Processor, 3593.65 MHz, 17-71-00 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,AMCR8,ABM,SSE4A,MASSE,3DNOWP,DBKP,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,IBPB,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache cpu0: 512KB 64b/line 8-way L2 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges cpu0: apic clock running at 99MHz cpu1 at mainbus0: apid 2 (application processor) cpu1: AMD Ryzen 7 3700X 8-Core Processor, 3593.27 MHz, 17-71-00 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,AMCR8,ABM,SSE4A,MASSE,3DNOWP,DBKP,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,IBPB,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu1: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache cpu1: 512KB 64b/line 8-way L2 cache cpu1: smt 0, core 2, package 0 cpu2 at mainbus0: apid 4 (application processor) cpu2: AMD Ryzen 7 3700X 8-Core Processor, 3593.26 MHz, 17-71-00 cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,AMCR8,ABM,SSE4A,MASSE,3DNOWP,DBKP,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,IBPB,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu2: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache cpu2: 512KB 64b/line 8-way L2 cache cpu2: smt 0, core 4, package 0 cpu3 at mainbus0: apid 6 (application processor) cpu3: AMD Ryzen 7 3700X 8-Core Processor, 3593.25 MHz, 17-71-00 cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,AMCR8,ABM,SSE4A,MASSE,3DNOWP,DBKP,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,IBPB,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu3: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache cpu3: 512KB 64b/line 8-way L2 cache cpu3: smt 0, core 6, package 0 cpu4 at mainbus0: apid 8 (application processor) cpu4: AMD Ryzen 7 3700X 8-Core Processor, 3593.27 MHz, 17-71-00 cpu4: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,AMCR8,ABM,SSE4A,MASSE,3DNOWP,DBKP,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,IBPB,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu4: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache cpu4: 512KB 64b/line 8-way L2 cache cpu4: smt 0, core 8, package 0 cpu5 at mainbus0: apid 10 (application processor) cpu5: AMD Ryzen 7 3700X 8-Core Processor, 3593.32 MHz, 17-71-00 cpu5: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,AMCR8,ABM,SSE4A,MASSE,3DNOWP,DBKP,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,IBPB,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu5: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache cpu5: 512KB 64b/line 8-way L2 cache cpu5: smt 0, core 10, package 0 cpu6 at mainbus0: apid 12 (application processor) cpu6: AMD Ryzen 7 3700X 8-Core Processor, 3593.34 MHz, 17-71-00 cpu6:
Re: [v3] amd64: simplify TSC sync testing
On Thu, 07 Jul 2022 14:02:35 +0900 (JST) YASUOKA Masahiko wrote: > Hello Scott, > > With the patch, my machine on ESXi it doesn't show any extra message. > > *Without* the patch, the machine shows > > % grep 'TSC.*skew' dmesg.current-tsc-debug > cpu1: disabling user TSC (skew=-2603) > cpu2: disabling user TSC (skew=-2959) > cpu3: disabling user TSC (skew=-3784) > % > > and monotonic time goes backward (the test in > https://marc.info/?l=openbsd-tech=161699406119704=2 failed) Oops, the link was wrong.. https://marc.info/?l=openbsd-tech=161657532610882=2 is the correct one. Also, with the patch, the test is passed. > dmesg of with the patch: > > OpenBSD 7.1-current (GENERIC.MP) #24: Thu Jul 7 10:09:32 JST 2022 > > yasu...@yasuoka-ob-c.tokyo.iiji.jp:/source/yasuoka/head/git/src/sys/arch/amd64/compile/GENERIC.MP > real mem = 2113290240 (2015MB) > avail mem = 2031927296 (1937MB) > random: good seed from bootblocks > mpath0 at root > scsibus0 at mpath0: 256 targets > mainbus0 at root > bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xff7401f (161 entries) > bios0: vendor VMware, Inc. version "VMW71.00V.16460286.B64.2006250725" date > 06/25/2020 > bios0: VMware, Inc. VMware7,1 > acpi0 at bios0: ACPI 4.0 > acpi0: sleep states S0 S1 S4 S5 > acpi0: tables DSDT SRAT FACP APIC MCFG HPET WAET WSMT > acpi0: wakeup devices PCI0(S3) USB_(S1) P2P0(S3) S1F0(S3) S2F0(S3) S8F0(S3) > S16F(S3) S17F(S3) S18F(S3) S22F(S3) S23F(S3) S24F(S3) S25F(S3) PE40(S3) > S1F0(S3) PE41(S3) [...] > acpitimer0 at acpi0: 3579545 Hz, 24 bits > acpimadt0 at acpi0 addr 0xfee0: PC-AT compat > cpu0 at mainbus0: apid 0 (boot processor) > cpu0: Intel(R) Xeon(R) Silver 4210R CPU @ 2.40GHz, 2393.81 MHz, 06-55-07 > cpu0: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,AVX512CD,AVX512BW,AVX512VL,PKU,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES > cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 1MB 64b/line > 16-way L2 cache, 13MB 64b/line 11-way L3 cache > cpu0: smt 0, core 0, package 0 > mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges > cpu0: apic clock running at 65MHz > cpu1 at mainbus0: apid 1 (application processor) > cpu1: Intel(R) Xeon(R) Silver 4210R CPU @ 2.40GHz, 2393.61 MHz, 06-55-07 > cpu1: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,AVX512CD,AVX512BW,AVX512VL,PKU,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES > cpu1: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 1MB 64b/line > 16-way L2 cache, 13MB 64b/line 11-way L3 cache > cpu1: smt 0, core 1, package 0 > cpu2 at mainbus0: apid 2 (application processor) > cpu2: Intel(R) Xeon(R) Silver 4210R CPU @ 2.40GHz, 2393.62 MHz, 06-55-07 > cpu2: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,AVX512CD,AVX512BW,AVX512VL,PKU,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES > cpu2: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 1MB 64b/line > 16-way L2 cache, 13MB 64b/line 11-way L3 cache > cpu2: smt 0, core 2, package 0 > cpu3 at mainbus0: apid 3 (application processor) > cpu3: Intel(R) Xeon(R) Silver 4210R CPU @ 2.40GHz, 2393.62 MHz, 06-55-07 > cpu3: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,AVX512CD,AVX512BW,AVX512VL,PKU,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES > cpu3: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 1MB 64b/line > 16-way L2 cache, 13MB 64b/line 11-way L3 cache > cpu3: smt 0, core 3, package 0 > ioapic0 at mainbus0: apid 4 pa 0xfec0, version 20, 24 pins, remapped > acpimcfg0 at acpi0 > acpimcfg0: addr 0xe000, bus 0-127 > acpihpet0 at acpi0: 14318179 Hz > acpiprt0 at acpi0: bus 0 (PCI0) > acpipci0 at acpi0 PCI0: 0x
Re: [v3] amd64: simplify TSC sync testing
Hello Scott, With the patch, my machine on ESXi it doesn't show any extra message. *Without* the patch, the machine shows % grep 'TSC.*skew' dmesg.current-tsc-debug cpu1: disabling user TSC (skew=-2603) cpu2: disabling user TSC (skew=-2959) cpu3: disabling user TSC (skew=-3784) % and monotonic time goes backward (the test in https://marc.info/?l=openbsd-tech=161699406119704=2 failed) dmesg of with the patch: OpenBSD 7.1-current (GENERIC.MP) #24: Thu Jul 7 10:09:32 JST 2022 yasu...@yasuoka-ob-c.tokyo.iiji.jp:/source/yasuoka/head/git/src/sys/arch/amd64/compile/GENERIC.MP real mem = 2113290240 (2015MB) avail mem = 2031927296 (1937MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xff7401f (161 entries) bios0: vendor VMware, Inc. version "VMW71.00V.16460286.B64.2006250725" date 06/25/2020 bios0: VMware, Inc. VMware7,1 acpi0 at bios0: ACPI 4.0 acpi0: sleep states S0 S1 S4 S5 acpi0: tables DSDT SRAT FACP APIC MCFG HPET WAET WSMT acpi0: wakeup devices PCI0(S3) USB_(S1) P2P0(S3) S1F0(S3) S2F0(S3) S8F0(S3) S16F(S3) S17F(S3) S18F(S3) S22F(S3) S23F(S3) S24F(S3) S25F(S3) PE40(S3) S1F0(S3) PE41(S3) [...] acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Xeon(R) Silver 4210R CPU @ 2.40GHz, 2393.81 MHz, 06-55-07 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,AVX512CD,AVX512BW,AVX512VL,PKU,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 1MB 64b/line 16-way L2 cache, 13MB 64b/line 11-way L3 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges cpu0: apic clock running at 65MHz cpu1 at mainbus0: apid 1 (application processor) cpu1: Intel(R) Xeon(R) Silver 4210R CPU @ 2.40GHz, 2393.61 MHz, 06-55-07 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,AVX512CD,AVX512BW,AVX512VL,PKU,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu1: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 1MB 64b/line 16-way L2 cache, 13MB 64b/line 11-way L3 cache cpu1: smt 0, core 1, package 0 cpu2 at mainbus0: apid 2 (application processor) cpu2: Intel(R) Xeon(R) Silver 4210R CPU @ 2.40GHz, 2393.62 MHz, 06-55-07 cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,AVX512CD,AVX512BW,AVX512VL,PKU,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu2: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 1MB 64b/line 16-way L2 cache, 13MB 64b/line 11-way L3 cache cpu2: smt 0, core 2, package 0 cpu3 at mainbus0: apid 3 (application processor) cpu3: Intel(R) Xeon(R) Silver 4210R CPU @ 2.40GHz, 2393.62 MHz, 06-55-07 cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,AVX512CD,AVX512BW,AVX512VL,PKU,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu3: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 1MB 64b/line 16-way L2 cache, 13MB 64b/line 11-way L3 cache cpu3: smt 0, core 3, package 0 ioapic0 at mainbus0: apid 4 pa 0xfec0, version 20, 24 pins, remapped acpimcfg0 at acpi0 acpimcfg0: addr 0xe000, bus 0-127 acpihpet0 at acpi0: 14318179 Hz acpiprt0 at acpi0: bus 0 (PCI0) acpipci0 at acpi0 PCI0: 0x 0x0011 0x0001 acpicmos0 at acpi0 "PNP0A05" at acpi0 not configured com0 at acpi0 COMA addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo com0: console acpiac0 at acpi0: AC unit online acpicpu0 at acpi0: C1(@1 halt!) acpicpu1 at acpi0: C1(@1 halt!) acpicpu2 at acpi0: C1(@1 halt!) acpicpu3 at acpi0: C1(@1 halt!) pvbus0 at mainbus0: VMware vmt0 at pvbus0 pci0 at
Re: [v3] amd64: simplify TSC sync testing
On 7/7/22 11:29, Scott Cheloha wrote:
On Thu, Jul 07, 2022 at 11:13:14AM +0900, Yuichiro NAITO wrote:
I had another try for the v3 patch that runs on FreeBSD bhyve.
In this case, I need the following patch for TSC to be chosen as a timecounter.
diff --git a/sys/arch/amd64/amd64/tsc.c b/sys/arch/amd64/amd64/tsc.c
index c4d3acda8c7..ab34df4463c 100644
--- a/sys/arch/amd64/amd64/tsc.c
+++ b/sys/arch/amd64/amd64/tsc.c
@@ -168,7 +168,7 @@ measure_tsc_freq(struct timecounter *tc)
min_freq = ULLONG_MAX;
- delay_usec = 10;
+ delay_usec = 1;
for (i = 0; i < 3; i++) {
s = intr_disable();
This is a separate issue from TSC synchronization.
I'll describe it in an another mail.
What are we using for delay_func when this function runs
on cpu0?
`i8254_delay` is used in the `measure_tsc_freq` execution time.
With this patch, Scott's v3 patch works fine on FreeBSD bhyve.
Here is dmesg shown in my OpenBSD.
This is an Intel machine. Does byhve not pass the TSC frequency
through the MSR? Or is there something special we need to do if
we're a guest to determine the frequency?
I'm not sure TSC frequency is offered via MSR.
`tsc_freq_cpuid` gets TSC frequency from CPUID 0x15.
It is disabled by bhyve.
So, OpenBSD kernel measures TSC frequency in `measure_tsc_freq` function.
But i8254_delay error is relatively large in this environment.
--
Yuichiro NAITO (naito.yuich...@gmail.com)
Re: [v3] amd64: simplify TSC sync testing
On Thu, Jul 07, 2022 at 11:13:14AM +0900, Yuichiro NAITO wrote:
> I had another try for the v3 patch that runs on FreeBSD bhyve.
> In this case, I need the following patch for TSC to be chosen as a
> timecounter.
>
> diff --git a/sys/arch/amd64/amd64/tsc.c b/sys/arch/amd64/amd64/tsc.c
> index c4d3acda8c7..ab34df4463c 100644
> --- a/sys/arch/amd64/amd64/tsc.c
> +++ b/sys/arch/amd64/amd64/tsc.c
> @@ -168,7 +168,7 @@ measure_tsc_freq(struct timecounter *tc)
>
> min_freq = ULLONG_MAX;
>
> - delay_usec = 10;
> + delay_usec = 1;
> for (i = 0; i < 3; i++) {
> s = intr_disable();
>
> This is a separate issue from TSC synchronization.
> I'll describe it in an another mail.
What are we using for delay_func when this function runs
on cpu0?
> With this patch, Scott's v3 patch works fine on FreeBSD bhyve.
> Here is dmesg shown in my OpenBSD.
This is an Intel machine. Does byhve not pass the TSC frequency
through the MSR? Or is there something special we need to do if
we're a guest to determine the frequency?
> OpenBSD 7.1-current (GENERIC.MP) #88: Wed Jul 6 16:11:53 JST 2022
> yuich...@opentest.soum.co.jp:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> real mem = 4278190080 (4080MB)
> avail mem = 4134338560 (3942MB)
> warning: no entropy supplied by boot loader
> random: boothowto does not indicate good seed
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 2.6 @ 0xf101f (14 entries)
> bios0: vendor BHYVE version "13.0" date 11/10/2020
> bios0: FreeBSD BHYVE
> acpi0 at bios0: ACPI 5.1
> acpi0: sleep states S5
> acpi0: tables DSDT APIC FACP HPET MCFG
> acpi0: wakeup devices
> acpitimer0 at acpi0: 3579545 Hz, 32 bits
> acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: Intel(R) Core(TM) i7-9700 CPU @ 3.00GHz, 3000.65 MHz, 06-9e-0d
> cpu0:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,PBE,SSE3,PCLMUL,DTES64,DS-CPL,SSSE3,SDBG,FMA3,CX16,xTPR,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,SMAP,MD_CLEAR,ARAT,XSAVEOPT,MELTDOWN
> cpu0: 256KB 64b/line 8-way L2 cache
> cpu0: smt 0, core 0, package 0
> mtrr: CPU supports MTRRs but not enabled by BIOS
> cpu0: apic clock running at 134MHz
> cpu1 at mainbus0: apid 1 (application processor)
> cpu1: Intel(R) Core(TM) i7-9700 CPU @ 3.00GHz, 3018.22 MHz, 06-9e-0d
> cpu1:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,PBE,SSE3,PCLMUL,DTES64,DS-CPL,SSSE3,SDBG,FMA3,CX16,xTPR,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,SMAP,MD_CLEAR,ARAT,XSAVEOPT,MELTDOWN
> cpu1: 256KB 64b/line 8-way L2 cache
> cpu1: smt 0, core 0, package 1
> cpu2 at mainbus0: apid 2 (application processor)
> cpu2: Intel(R) Core(TM) i7-9700 CPU @ 3.00GHz, 3018.27 MHz, 06-9e-0d
> cpu2:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,PBE,SSE3,PCLMUL,DTES64,DS-CPL,SSSE3,SDBG,FMA3,CX16,xTPR,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,SMAP,MD_CLEAR,ARAT,XSAVEOPT,MELTDOWN
> cpu2: 256KB 64b/line 8-way L2 cache
> cpu2: smt 0, core 0, package 2
> cpu3 at mainbus0: apid 3 (application processor)
> cpu3: Intel(R) Core(TM) i7-9700 CPU @ 3.00GHz, 3018.05 MHz, 06-9e-0d
> cpu3:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,PBE,SSE3,PCLMUL,DTES64,DS-CPL,SSSE3,SDBG,FMA3,CX16,xTPR,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,SMAP,MD_CLEAR,ARAT,XSAVEOPT,MELTDOWN
> cpu3: 256KB 64b/line 8-way L2 cache
> cpu3: smt 0, core 0, package 3
> ioapic0 at mainbus0: apid 0 pa 0xfec0, version 11, 32 pins
> acpihpet0 at acpi0: 16777216 Hz
> acpimcfg0 at acpi0
> acpimcfg0: addr 0xe000, bus 0-255
> acpiprt0 at acpi0: bus 0 (PC00)
> acpipci0 at acpi0 PC00
> com0 at acpi0 COM1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo
> com0: console
> com1 at acpi0 COM2 addr 0x2f8/0x8 irq 3: ns16550a, 16 byte fifo
> com2 at acpi0 COM3 addr 0x3e8/0x8 irq 4: ns16550a, 16 byte fifo
> com3 at acpi0 COM4 addr 0x2e8/0x8 irq 3: ns16550a, 16 byte fifo
> acpicmos0 at acpi0
> "Bhyve_V_Gen_Counter_V1" at acpi0 not configured
> cpu0: using VERW MDS workaround
> pvbus0 at mainbus0: bhyve
> pci0 at mainbus0 bus 0
> pchb0 at pci0 dev 0 function 0 unknown vendor 0x1275 product 0x1275 rev 0x00
> pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
> virtio0 at pci0 dev 2 function 0 "Qumranet Virtio Storage" rev 0x00
>
Re: [v3] amd64: simplify TSC sync testing
I had another try for the v3 patch that runs on FreeBSD bhyve.
In this case, I need the following patch for TSC to be chosen as a timecounter.
diff --git a/sys/arch/amd64/amd64/tsc.c b/sys/arch/amd64/amd64/tsc.c
index c4d3acda8c7..ab34df4463c 100644
--- a/sys/arch/amd64/amd64/tsc.c
+++ b/sys/arch/amd64/amd64/tsc.c
@@ -168,7 +168,7 @@ measure_tsc_freq(struct timecounter *tc)
min_freq = ULLONG_MAX;
- delay_usec = 10;
+ delay_usec = 1;
for (i = 0; i < 3; i++) {
s = intr_disable();
This is a separate issue from TSC synchronization.
I'll describe it in an another mail.
With this patch, Scott's v3 patch works fine on FreeBSD bhyve.
Here is dmesg shown in my OpenBSD.
OpenBSD 7.1-current (GENERIC.MP) #88: Wed Jul 6 16:11:53 JST 2022
yuich...@opentest.soum.co.jp:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 4278190080 (4080MB)
avail mem = 4134338560 (3942MB)
warning: no entropy supplied by boot loader
random: boothowto does not indicate good seed
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.6 @ 0xf101f (14 entries)
bios0: vendor BHYVE version "13.0" date 11/10/2020
bios0: FreeBSD BHYVE
acpi0 at bios0: ACPI 5.1
acpi0: sleep states S5
acpi0: tables DSDT APIC FACP HPET MCFG
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 32 bits
acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Core(TM) i7-9700 CPU @ 3.00GHz, 3000.65 MHz, 06-9e-0d
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,PBE,SSE3,PCLMUL,DTES64,DS-CPL,SSSE3,SDBG,FMA3,CX16,xTPR,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,SMAP,MD_CLEAR,ARAT,XSAVEOPT,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: CPU supports MTRRs but not enabled by BIOS
cpu0: apic clock running at 134MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Intel(R) Core(TM) i7-9700 CPU @ 3.00GHz, 3018.22 MHz, 06-9e-0d
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,PBE,SSE3,PCLMUL,DTES64,DS-CPL,SSSE3,SDBG,FMA3,CX16,xTPR,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,SMAP,MD_CLEAR,ARAT,XSAVEOPT,MELTDOWN
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 0, core 0, package 1
cpu2 at mainbus0: apid 2 (application processor)
cpu2: Intel(R) Core(TM) i7-9700 CPU @ 3.00GHz, 3018.27 MHz, 06-9e-0d
cpu2:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,PBE,SSE3,PCLMUL,DTES64,DS-CPL,SSSE3,SDBG,FMA3,CX16,xTPR,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,SMAP,MD_CLEAR,ARAT,XSAVEOPT,MELTDOWN
cpu2: 256KB 64b/line 8-way L2 cache
cpu2: smt 0, core 0, package 2
cpu3 at mainbus0: apid 3 (application processor)
cpu3: Intel(R) Core(TM) i7-9700 CPU @ 3.00GHz, 3018.05 MHz, 06-9e-0d
cpu3:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,PBE,SSE3,PCLMUL,DTES64,DS-CPL,SSSE3,SDBG,FMA3,CX16,xTPR,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,SMAP,MD_CLEAR,ARAT,XSAVEOPT,MELTDOWN
cpu3: 256KB 64b/line 8-way L2 cache
cpu3: smt 0, core 0, package 3
ioapic0 at mainbus0: apid 0 pa 0xfec0, version 11, 32 pins
acpihpet0 at acpi0: 16777216 Hz
acpimcfg0 at acpi0
acpimcfg0: addr 0xe000, bus 0-255
acpiprt0 at acpi0: bus 0 (PC00)
acpipci0 at acpi0 PC00
com0 at acpi0 COM1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo
com0: console
com1 at acpi0 COM2 addr 0x2f8/0x8 irq 3: ns16550a, 16 byte fifo
com2 at acpi0 COM3 addr 0x3e8/0x8 irq 4: ns16550a, 16 byte fifo
com3 at acpi0 COM4 addr 0x2e8/0x8 irq 3: ns16550a, 16 byte fifo
acpicmos0 at acpi0
"Bhyve_V_Gen_Counter_V1" at acpi0 not configured
cpu0: using VERW MDS workaround
pvbus0 at mainbus0: bhyve
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 unknown vendor 0x1275 product 0x1275 rev 0x00
pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
virtio0 at pci0 dev 2 function 0 "Qumranet Virtio Storage" rev 0x00
vioblk0 at virtio0
scsibus1 at vioblk0: 1 targets
sd0 at scsibus1 targ 0 lun 0:
sd0: 20480MB, 512 bytes/sector, 41943040 sectors
virtio0: msix shared
ahci0 at pci0 dev 3 function 0 "Intel 82801H AHCI" rev 0x00: msi, AHCI 1.3
ahci0: port 0: 6.0Gb/s
scsibus2 at ahci0: 32 targets
cd0 at scsibus2 targ 0 lun 0: removable
virtio1 at pci0 dev 4 function 0 "Qumranet Virtio Network" rev 0x00
vio0 at virtio1: address 00:a0:98:db:89:86
virtio1: msix shared
isa0 at
Re: [v3] amd64: simplify TSC sync testing
On Wed, Jul 06, 2022 at 01:58:51PM -0700, Mike Larkin wrote: > On Wed, Jul 06, 2022 at 11:48:41AM -0500, Scott Cheloha wrote: > > > On Jul 6, 2022, at 11:36 AM, Mike Larkin wrote: > > > > > > On Tue, Jul 05, 2022 at 07:16:26PM -0500, Scott Cheloha wrote: > > >> On Tue, Jul 05, 2022 at 01:38:32PM -0700, Mike Larkin wrote: > > >>> On Mon, Jul 04, 2022 at 09:06:55PM -0500, Scott Cheloha wrote: > > > > [...] > > >>> > > >>> Here's the output from a 4 socket 80 thread machine. > > >> > > >> Oh nice. I think this is the biggest machine we've tried so far. > > >> > > >>> kern.timecounter reports tsc after boot. > > >> > > >> Excellent. > > >> > > >>> Looks like this machine doesn't have the adjust MSR? > > >> > > >> IA32_TSC_ADJUST first appears in the Intel SDM Vol. 3 some time in > > >> 2011 or 2012. I can't find the exact revision. > > >> > > >> (I really wish there was a comprehensive version history for this sort > > >> of thing, i.e. this MSR first appeared in the blah-blah uarch, this > > >> instruction is available on all uarchs after yada-yada, etc.) > > >> > > >> There are apparently several versions of the E7-4870 in the E7 > > >> "family". If your CPU predates that, or launched 2012-2014, the MSR > > >> may not have made the cut. > > >> > > >> An aside: I cannot find any evidence of AMD supporting this MSR in any > > >> processor. It would be really, really nice if they did. If you (or > > >> anyone reading) knows anything about this, or whether they have an > > >> equivalent MSR, shout it out. > > >> > > >>> Other than that, machine seems stable. > > >> > > >> Good, glad to hear it. Thank you for testing. > > >> > > >> Has this machine had issues using the TSC on -current in the past? > > >> > > >> (If you have the time) what does the dmesg look like on the -current > > >> kernel with TSC_DEBUG enabled? > > > > > > Looks like you enabled TSC_DEBUG in your diff, so what I sent you is what > > > you > > > are asking for...? > > > > No, I mean on the -current *unpatched* kernel. Sorry if that wasn't > > clear. > > > > Our -current kernel prints more detailed information if TSC_DEBUG > > is enabled. In particular, I'm curious if the unpatched kernel > > detects any skew or drift on your machine, and if so, how much. > > > > here you go. I didnt run with all 80 cpus since -current doesnt have my > " > 64 cpus" diff, but I think this is what you're after in any case. Yes! This is what I was looking for, thanks. > cpu0: TSC skew=0 observed drift=0 > cpu1: TSC skew=112 observed drift=0 > cpu2: TSC skew=102 observed drift=0 > cpu3: TSC skew=-134 observed drift=0 > cpu4: TSC skew=4 observed drift=0 > cpu5: TSC skew=68 observed drift=0 > cpu6: TSC skew=22 observed drift=0 > cpu7: TSC skew=-52 observed drift=0 > cpu8: TSC skew=8 observed drift=0 > cpu9: TSC skew=-18 observed drift=0 > cpu10: TSC skew=10 observed drift=0 > cpu11: TSC skew=76 observed drift=0 > cpu12: TSC skew=-2 observed drift=0 > cpu13: TSC skew=-4 observed drift=0 > cpu14: TSC skew=-2 observed drift=0 > cpu15: TSC skew=-28 observed drift=0 > cpu16: TSC skew=6 observed drift=0 > cpu17: TSC skew=-8 observed drift=0 > cpu18: TSC skew=0 observed drift=0 > cpu19: TSC skew=-32 observed drift=0 > cpu20: TSC skew=0 observed drift=0 > cpu21: TSC skew=-26 observed drift=0 > cpu22: TSC skew=0 observed drift=0 > cpu23: TSC skew=22 observed drift=0 > cpu24: TSC skew=-12 observed drift=0 > cpu25: TSC skew=-14 observed drift=0 > cpu26: TSC skew=76 observed drift=0 > cpu27: TSC skew=-64 observed drift=0 > cpu28: TSC skew=-2 observed drift=0 > cpu29: TSC skew=34 observed drift=0 > cpu30: TSC skew=22 observed drift=0 > cpu31: TSC skew=-58 observed drift=0 > cpu32: TSC skew=-2 observed drift=0 > cpu33: TSC skew=6 observed drift=0 > cpu34: TSC skew=46 observed drift=0 > cpu35: TSC skew=20 observed drift=0 > cpu36: TSC skew=34 observed drift=0 > cpu37: TSC skew=-8 observed drift=0 > cpu38: TSC skew=48 observed drift=0 > cpu39: TSC skew=-10 observed drift=0 > cpu40: TSC skew=0 observed drift=0 > cpu41: TSC skew=72 observed drift=0 > cpu42: TSC skew=2 observed drift=0 > cpu43: TSC skew=-46 observed drift=0 > cpu44: TSC skew=-2 observed drift=0 > cpu45: TSC skew=-14 observed drift=0 > cpu46: TSC skew=-2 observed drift=0 > cpu47: TSC skew=-32 observed drift=0 > cpu48: TSC skew=12 observed drift=0 > cpu49: TSC skew=-16 observed drift=0 > cpu50: TSC skew=84 observed drift=0 > cpu51: TSC skew=-44 observed drift=0 > cpu52: TSC skew=-4 observed drift=0 > cpu53: TSC skew=4 observed drift=0 > cpu54: TSC skew=16 observed drift=0 > cpu55: TSC skew=-56 observed drift=0 > cpu56: TSC skew=-10 observed drift=0 > cpu57: TSC skew=6 observed drift=0 > cpu58: TSC skew=6 observed drift=0 > cpu59: TSC skew=-40 observed drift=0 > cpu60: TSC skew=-4 observed drift=0 > cpu61: TSC skew=-6 observed drift=0 > cpu62: TSC skew=74 observed drift=0 > cpu63: TSC skew=-48 observed drift=0
Re: [v3] amd64: simplify TSC sync testing
On Wed, Jul 06, 2022 at 08:20:05PM -0400, Mohamed Aslan wrote: > > First, you need to update to the latest firmware. Maybe they already > > fixed the problem. I don't see any mention of the TSC in the BIOS > > changelog for the e495 but maybe you'll get lucky. > > > > Second, if they haven't fixed the problem with the latest firmware, I > > recommend you reach out to Lenovo and report the problem. > > > > Lenovo seem to have been sympathetic to reports about TSC desync in > > the past on other models and issued firmware fixes. For example, > > the v1.28 firmware for the ThinkPad A485 contained a fix for what > > I assume is a very similar problem to the one you're having: > > > > https://download.lenovo.com/pccbbs/mobiles/r0wuj65wd.txt > > > > And this forum post, for example, got some response from Lenovo staff: > > > > https://forums.lenovo.com/t5/ThinkPad-T400-T500-and-newer-T-series-Laptops/T14s-G1-AMD-TSC-clock-unusable/m-p/5070296?page=1 > > > > So, open a post for your model and cite the other posts. > > > > They might not be sympathetic to the fact that you're seeing the issue > > on OpenBSD. If that's a problem you should be able to reproduce the > > problem with a recent Linux kernel. The Linux kernel runs a similar > > sync test during boot and will complain if the TSCs are not > > synchronized. > > > > Honestly, to save time you may want to just boot up a supported Linux > > distribution and grab the error message before you ask for support. > > > > I can confirm that this is also the case with Linux. This is the > output of dmesg on Void Linux: > > [0.00] tsc: Fast TSC calibration using PIT > [0.00] tsc: Detected 2096.114 MHz processor > ... > ... > [1.314252] TSC synchronization [CPU#0 -> CPU#1]: > [1.314252] Measured 6615806646 cycles TSC warp between CPUs, turning off > TSC clock. > [1.314252] tsc: Marking TSC unstable due to check_tsc_sync_source failed > [1.314397] #2 #3 #4 #5 #6 #7 This is good news. My code isn't the only code finding a problem :) > > Not sure if Void is a Lenovo supported Linux distribution, still > though I think it's worth reporting. Probably not. Your laptop may not even be "Linux certified", but it's worth reporting all the same.
Re: [v3] amd64: simplify TSC sync testing
> First, you need to update to the latest firmware. Maybe they already > fixed the problem. I don't see any mention of the TSC in the BIOS > changelog for the e495 but maybe you'll get lucky. > > Second, if they haven't fixed the problem with the latest firmware, I > recommend you reach out to Lenovo and report the problem. > > Lenovo seem to have been sympathetic to reports about TSC desync in > the past on other models and issued firmware fixes. For example, > the v1.28 firmware for the ThinkPad A485 contained a fix for what > I assume is a very similar problem to the one you're having: > > https://download.lenovo.com/pccbbs/mobiles/r0wuj65wd.txt > > And this forum post, for example, got some response from Lenovo staff: > > https://forums.lenovo.com/t5/ThinkPad-T400-T500-and-newer-T-series-Laptops/T14s-G1-AMD-TSC-clock-unusable/m-p/5070296?page=1 > > So, open a post for your model and cite the other posts. > > They might not be sympathetic to the fact that you're seeing the issue > on OpenBSD. If that's a problem you should be able to reproduce the > problem with a recent Linux kernel. The Linux kernel runs a similar > sync test during boot and will complain if the TSCs are not > synchronized. > > Honestly, to save time you may want to just boot up a supported Linux > distribution and grab the error message before you ask for support. > I can confirm that this is also the case with Linux. This is the output of dmesg on Void Linux: [0.00] tsc: Fast TSC calibration using PIT [0.00] tsc: Detected 2096.114 MHz processor ... ... [1.314252] TSC synchronization [CPU#0 -> CPU#1]: [1.314252] Measured 6615806646 cycles TSC warp between CPUs, turning off TSC clock. [1.314252] tsc: Marking TSC unstable due to check_tsc_sync_source failed [1.314397] #2 #3 #4 #5 #6 #7 Not sure if Void is a Lenovo supported Linux distribution, still though I think it's worth reporting. Thanks
Re: [v3] amd64: simplify TSC sync testing
On Wed, Jul 06, 2022 at 11:48:41AM -0500, Scott Cheloha wrote: > > On Jul 6, 2022, at 11:36 AM, Mike Larkin wrote: > > > > On Tue, Jul 05, 2022 at 07:16:26PM -0500, Scott Cheloha wrote: > >> On Tue, Jul 05, 2022 at 01:38:32PM -0700, Mike Larkin wrote: > >>> On Mon, Jul 04, 2022 at 09:06:55PM -0500, Scott Cheloha wrote: > > [...] > >>> > >>> Here's the output from a 4 socket 80 thread machine. > >> > >> Oh nice. I think this is the biggest machine we've tried so far. > >> > >>> kern.timecounter reports tsc after boot. > >> > >> Excellent. > >> > >>> Looks like this machine doesn't have the adjust MSR? > >> > >> IA32_TSC_ADJUST first appears in the Intel SDM Vol. 3 some time in > >> 2011 or 2012. I can't find the exact revision. > >> > >> (I really wish there was a comprehensive version history for this sort > >> of thing, i.e. this MSR first appeared in the blah-blah uarch, this > >> instruction is available on all uarchs after yada-yada, etc.) > >> > >> There are apparently several versions of the E7-4870 in the E7 > >> "family". If your CPU predates that, or launched 2012-2014, the MSR > >> may not have made the cut. > >> > >> An aside: I cannot find any evidence of AMD supporting this MSR in any > >> processor. It would be really, really nice if they did. If you (or > >> anyone reading) knows anything about this, or whether they have an > >> equivalent MSR, shout it out. > >> > >>> Other than that, machine seems stable. > >> > >> Good, glad to hear it. Thank you for testing. > >> > >> Has this machine had issues using the TSC on -current in the past? > >> > >> (If you have the time) what does the dmesg look like on the -current > >> kernel with TSC_DEBUG enabled? > > > > Looks like you enabled TSC_DEBUG in your diff, so what I sent you is what > > you > > are asking for...? > > No, I mean on the -current *unpatched* kernel. Sorry if that wasn't > clear. > > Our -current kernel prints more detailed information if TSC_DEBUG > is enabled. In particular, I'm curious if the unpatched kernel > detects any skew or drift on your machine, and if so, how much. > here you go. I didnt run with all 80 cpus since -current doesnt have my " > 64 cpus" diff, but I think this is what you're after in any case. -ml cpu0: TSC skew=0 observed drift=0 cpu1: TSC skew=112 observed drift=0 cpu2: TSC skew=102 observed drift=0 cpu3: TSC skew=-134 observed drift=0 cpu4: TSC skew=4 observed drift=0 cpu5: TSC skew=68 observed drift=0 cpu6: TSC skew=22 observed drift=0 cpu7: TSC skew=-52 observed drift=0 cpu8: TSC skew=8 observed drift=0 cpu9: TSC skew=-18 observed drift=0 cpu10: TSC skew=10 observed drift=0 cpu11: TSC skew=76 observed drift=0 cpu12: TSC skew=-2 observed drift=0 cpu13: TSC skew=-4 observed drift=0 cpu14: TSC skew=-2 observed drift=0 cpu15: TSC skew=-28 observed drift=0 cpu16: TSC skew=6 observed drift=0 cpu17: TSC skew=-8 observed drift=0 cpu18: TSC skew=0 observed drift=0 cpu19: TSC skew=-32 observed drift=0 cpu20: TSC skew=0 observed drift=0 cpu21: TSC skew=-26 observed drift=0 cpu22: TSC skew=0 observed drift=0 cpu23: TSC skew=22 observed drift=0 cpu24: TSC skew=-12 observed drift=0 cpu25: TSC skew=-14 observed drift=0 cpu26: TSC skew=76 observed drift=0 cpu27: TSC skew=-64 observed drift=0 cpu28: TSC skew=-2 observed drift=0 cpu29: TSC skew=34 observed drift=0 cpu30: TSC skew=22 observed drift=0 cpu31: TSC skew=-58 observed drift=0 cpu32: TSC skew=-2 observed drift=0 cpu33: TSC skew=6 observed drift=0 cpu34: TSC skew=46 observed drift=0 cpu35: TSC skew=20 observed drift=0 cpu36: TSC skew=34 observed drift=0 cpu37: TSC skew=-8 observed drift=0 cpu38: TSC skew=48 observed drift=0 cpu39: TSC skew=-10 observed drift=0 cpu40: TSC skew=0 observed drift=0 cpu41: TSC skew=72 observed drift=0 cpu42: TSC skew=2 observed drift=0 cpu43: TSC skew=-46 observed drift=0 cpu44: TSC skew=-2 observed drift=0 cpu45: TSC skew=-14 observed drift=0 cpu46: TSC skew=-2 observed drift=0 cpu47: TSC skew=-32 observed drift=0 cpu48: TSC skew=12 observed drift=0 cpu49: TSC skew=-16 observed drift=0 cpu50: TSC skew=84 observed drift=0 cpu51: TSC skew=-44 observed drift=0 cpu52: TSC skew=-4 observed drift=0 cpu53: TSC skew=4 observed drift=0 cpu54: TSC skew=16 observed drift=0 cpu55: TSC skew=-56 observed drift=0 cpu56: TSC skew=-10 observed drift=0 cpu57: TSC skew=6 observed drift=0 cpu58: TSC skew=6 observed drift=0 cpu59: TSC skew=-40 observed drift=0 cpu60: TSC skew=-4 observed drift=0 cpu61: TSC skew=-6 observed drift=0 cpu62: TSC skew=74 observed drift=0 cpu63: TSC skew=-48 observed drift=0
Re: [v3] amd64: simplify TSC sync testing
> On Jul 6, 2022, at 10:04 AM, Christian Weisgerber wrote:
>
> Scott Cheloha:
>
>>> kern.timecounter.tick=1
>>> kern.timecounter.timestepwarnings=0
>>> kern.timecounter.hardware=i8254
>>> kern.timecounter.choice=i8254(0) tsc(-1000) acpihpet0(1000) acpitimer0(1000)
>>
>> This is expected behavior with the patch.
>>
>> cpu0's TSC is way out of sync with every
>> other CPU's TSC, so the TSC is marked
>> as a bad timecounter and a different one is
>> chosen.
>
> Shouldn't it pick acpihpet0 then?
It depends on the order the timecounters are installed.
If acpihpet0 is already installed before we degrade the
TSC's .quality value then the timecounter subsystem won't
switch to it when we install the next counter because it
assumes .quality values cannot change on the fly (a
reasonable assumption).
We don't yet have a tc_detach(9) function that uninstalls
a timecounter cleanly and chooses the next best counter
available.
This is something I want to add in a future patch. FreeBSD
has something similar. It may be called "tc_ban", iirc.
The alternative is to wait until we've tested synchronization
for every CPU before calling tc_init(9). This approach is
more annoying, though, as it requires additional state. We
would also still have the same problem when we resume from
suspend.
The dream is to be able to do something like this during the
sync test:
if (tsc_sync_test_failed) {
tc_detach(_timecounter);
tsc_timecounter.quality = -2000;
tc_init(_timecounter);
}
When we call tc_detach(9) the timecounter code would pick
the next best counter automagically.
Re: [v3] amd64: simplify TSC sync testing
> On Jul 6, 2022, at 11:36 AM, Mike Larkin wrote: > > On Tue, Jul 05, 2022 at 07:16:26PM -0500, Scott Cheloha wrote: >> On Tue, Jul 05, 2022 at 01:38:32PM -0700, Mike Larkin wrote: >>> On Mon, Jul 04, 2022 at 09:06:55PM -0500, Scott Cheloha wrote: [...] >>> >>> Here's the output from a 4 socket 80 thread machine. >> >> Oh nice. I think this is the biggest machine we've tried so far. >> >>> kern.timecounter reports tsc after boot. >> >> Excellent. >> >>> Looks like this machine doesn't have the adjust MSR? >> >> IA32_TSC_ADJUST first appears in the Intel SDM Vol. 3 some time in >> 2011 or 2012. I can't find the exact revision. >> >> (I really wish there was a comprehensive version history for this sort >> of thing, i.e. this MSR first appeared in the blah-blah uarch, this >> instruction is available on all uarchs after yada-yada, etc.) >> >> There are apparently several versions of the E7-4870 in the E7 >> "family". If your CPU predates that, or launched 2012-2014, the MSR >> may not have made the cut. >> >> An aside: I cannot find any evidence of AMD supporting this MSR in any >> processor. It would be really, really nice if they did. If you (or >> anyone reading) knows anything about this, or whether they have an >> equivalent MSR, shout it out. >> >>> Other than that, machine seems stable. >> >> Good, glad to hear it. Thank you for testing. >> >> Has this machine had issues using the TSC on -current in the past? >> >> (If you have the time) what does the dmesg look like on the -current >> kernel with TSC_DEBUG enabled? > > Looks like you enabled TSC_DEBUG in your diff, so what I sent you is what you > are asking for...? No, I mean on the -current *unpatched* kernel. Sorry if that wasn't clear. Our -current kernel prints more detailed information if TSC_DEBUG is enabled. In particular, I'm curious if the unpatched kernel detects any skew or drift on your machine, and if so, how much.
Re: [v3] amd64: simplify TSC sync testing
On Tue, Jul 05, 2022 at 07:16:26PM -0500, Scott Cheloha wrote: > On Tue, Jul 05, 2022 at 01:38:32PM -0700, Mike Larkin wrote: > > On Mon, Jul 04, 2022 at 09:06:55PM -0500, Scott Cheloha wrote: > > > > > > [...] > > > > Here's the output from a 4 socket 80 thread machine. > > Oh nice. I think this is the biggest machine we've tried so far. > > > kern.timecounter reports tsc after boot. > > Excellent. > > > Looks like this machine doesn't have the adjust MSR? > > IA32_TSC_ADJUST first appears in the Intel SDM Vol. 3 some time in > 2011 or 2012. I can't find the exact revision. > > (I really wish there was a comprehensive version history for this sort > of thing, i.e. this MSR first appeared in the blah-blah uarch, this > instruction is available on all uarchs after yada-yada, etc.) > > There are apparently several versions of the E7-4870 in the E7 > "family". If your CPU predates that, or launched 2012-2014, the MSR > may not have made the cut. > > An aside: I cannot find any evidence of AMD supporting this MSR in any > processor. It would be really, really nice if they did. If you (or > anyone reading) knows anything about this, or whether they have an > equivalent MSR, shout it out. > > > Other than that, machine seems stable. > > Good, glad to hear it. Thank you for testing. > > Has this machine had issues using the TSC on -current in the past? > > (If you have the time) what does the dmesg look like on the -current > kernel with TSC_DEBUG enabled? Looks like you enabled TSC_DEBUG in your diff, so what I sent you is what you are asking for...? -ml
Re: [v3] amd64: simplify TSC sync testing
On Wed, Jul 06, 2022 at 01:48:39AM -0400, Mohamed Aslan wrote: > > This is expected behavior with the patch. > > > > cpu0's TSC is way out of sync with every > > other CPU's TSC, so the TSC is marked > > as a bad timecounter and a different one is > > chosen. > > Yes, I can see. Just want to add that without your latest patch the > kernel chooses the TSC as clocksource, however only the *user* TSC > was disabled (cpu1: disabling user TSC (skew=-5028216492)). > > > Are you running the latest BIOS available > > for your machine? > > No, I don't think I am. First, you need to update to the latest firmware. Maybe they already fixed the problem. I don't see any mention of the TSC in the BIOS changelog for the e495 but maybe you'll get lucky. Second, if they haven't fixed the problem with the latest firmware, I recommend you reach out to Lenovo and report the problem. Lenovo seem to have been sympathetic to reports about TSC desync in the past on other models and issued firmware fixes. For example, the v1.28 firmware for the ThinkPad A485 contained a fix for what I assume is a very similar problem to the one you're having: https://download.lenovo.com/pccbbs/mobiles/r0wuj65wd.txt And this forum post, for example, got some response from Lenovo staff: https://forums.lenovo.com/t5/ThinkPad-T400-T500-and-newer-T-series-Laptops/T14s-G1-AMD-TSC-clock-unusable/m-p/5070296?page=1 So, open a post for your model and cite the other posts. They might not be sympathetic to the fact that you're seeing the issue on OpenBSD. If that's a problem you should be able to reproduce the problem with a recent Linux kernel. The Linux kernel runs a similar sync test during boot and will complain if the TSCs are not synchronized. Honestly, to save time you may want to just boot up a supported Linux distribution and grab the error message before you ask for support.
Re: [v3] amd64: simplify TSC sync testing
Scott Cheloha: > > kern.timecounter.tick=1 > > kern.timecounter.timestepwarnings=0 > > kern.timecounter.hardware=i8254 > > kern.timecounter.choice=i8254(0) tsc(-1000) acpihpet0(1000) acpitimer0(1000) > > This is expected behavior with the patch. > > cpu0's TSC is way out of sync with every > other CPU's TSC, so the TSC is marked > as a bad timecounter and a different one is > chosen. Shouldn't it pick acpihpet0 then? -- Christian "naddy" Weisgerber na...@mips.inka.de
Re: [v3] amd64: simplify TSC sync testing
> This is expected behavior with the patch. > > cpu0's TSC is way out of sync with every > other CPU's TSC, so the TSC is marked > as a bad timecounter and a different one is > chosen. Yes, I can see. Just want to add that without your latest patch the kernel chooses the TSC as clocksource, however only the *user* TSC was disabled (cpu1: disabling user TSC (skew=-5028216492)). > Are you running the latest BIOS available > for your machine? No, I don't think I am. Thanks
Re: [v3] amd64: simplify TSC sync testing
> On Jul 5, 2022, at 23:02, Mohamed Aslan wrote: > > Hi, > > Apologies. My bad, I applied the latest patch but booted into another > kernel with an earlier patch! > > Here's what I got with your latest patch: > > $ dmesg | grep 'tsc' > tsc: cpu0/cpu1: sync test round 1/2 failed > tsc: cpu0/cpu1: cpu0: 40162 lags 5112675666 cycles > tsc: cpu0/cpu2: sync test round 1/2 failed > tsc: cpu0/cpu2: cpu0: 18995 lags 5112675645 cycles > tsc: cpu0/cpu3: sync test round 1/2 failed > tsc: cpu0/cpu3: cpu0: 19136 lags 5112675645 cycles > tsc: cpu0/cpu4: sync test round 1/2 failed > tsc: cpu0/cpu4: cpu0: 19451 lags 5112675645 cycles > tsc: cpu0/cpu5: sync test round 1/2 failed > tsc: cpu0/cpu5: cpu0: 18625 lags 5112675645 cycles > tsc: cpu0/cpu6: sync test round 1/2 failed > tsc: cpu0/cpu6: cpu0: 18208 lags 5112675645 cycles > tsc: cpu0/cpu7: sync test round 1/2 failed > tsc: cpu0/cpu7: cpu0: 17739 lags 5112675645 cycles > > $ sysctl kern.timecounter > kern.timecounter.tick=1 > kern.timecounter.timestepwarnings=0 > kern.timecounter.hardware=i8254 > kern.timecounter.choice=i8254(0) tsc(-1000) acpihpet0(1000) acpitimer0(1000) This is expected behavior with the patch. cpu0's TSC is way out of sync with every other CPU's TSC, so the TSC is marked as a bad timecounter and a different one is chosen. Are you running the latest BIOS available for your machine?
Re: [SPAM] Re: [v3] amd64: simplify TSC sync testing
Hi, Apologies. My bad, I applied the latest patch but booted into another kernel with an earlier patch! Here's what I got with your latest patch: $ dmesg | grep 'tsc' tsc: cpu0/cpu1: sync test round 1/2 failed tsc: cpu0/cpu1: cpu0: 40162 lags 5112675666 cycles tsc: cpu0/cpu2: sync test round 1/2 failed tsc: cpu0/cpu2: cpu0: 18995 lags 5112675645 cycles tsc: cpu0/cpu3: sync test round 1/2 failed tsc: cpu0/cpu3: cpu0: 19136 lags 5112675645 cycles tsc: cpu0/cpu4: sync test round 1/2 failed tsc: cpu0/cpu4: cpu0: 19451 lags 5112675645 cycles tsc: cpu0/cpu5: sync test round 1/2 failed tsc: cpu0/cpu5: cpu0: 18625 lags 5112675645 cycles tsc: cpu0/cpu6: sync test round 1/2 failed tsc: cpu0/cpu6: cpu0: 18208 lags 5112675645 cycles tsc: cpu0/cpu7: sync test round 1/2 failed tsc: cpu0/cpu7: cpu0: 17739 lags 5112675645 cycles $ sysctl kern.timecounter kern.timecounter.tick=1 kern.timecounter.timestepwarnings=0 kern.timecounter.hardware=i8254 kern.timecounter.choice=i8254(0) tsc(-1000) acpihpet0(1000) acpitimer0(1000) OpenBSD 7.1-current (GENERIC.MP) #0: Tue Jul 5 23:33:01 EDT 2022 r...@lenovo.my.domain:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 7351164928 (7010MB) avail mem = 7111012352 (6781MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 3.1 @ 0xba4cb000 (58 entries) bios0: vendor LENOVO version "R11ET32W (1.12 )" date 12/23/2019 bios0: LENOVO 20NE0002US acpi0 at bios0: ACPI 5.0 acpi0: sleep states S0 S3 S4 S5 acpi0: tables DSDT FACP SSDT SSDT SSDT TPM2 SSDT MSDM SLIC BATB HPET APIC MCFG SBST WSMT VFCT IVRS SSDT CRAT CDIT FPDT SSDT SSDT SSDT BGRT UEFI acpi0: wakeup devices GPP0(S3) GPP1(S3) GPP2(S3) GPP3(S3) GPP4(S3) GPP5(S3) GPP6(S3) GP17(S3) XHC0(S3) XHC1(S3) GP18(S3) LID_(S3) SLPB(S3) acpitimer0 at acpi0: 3579545 Hz, 32 bits acpihpet0 at acpi0: 14318180 Hz acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: AMD Ryzen 5 3500U with Radeon Vega Mobile Gfx, 2096.33 MHz, 17-18-01 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA,IBPB,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu0: 32KB 64b/line 8-way D-cache, 64KB 64b/line 4-way I-cache, 512KB 64b/line 8-way L2 cache, 4MB 64b/line 16-way L3 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges cpu0: apic clock running at 24MHz cpu0: mwait min=64, max=64, C-substates=1.1, IBE cpu1 at mainbus0: apid 1 (application processor) tsc: cpu0/cpu1: sync test round 1/2 failed tsc: cpu0/cpu1: cpu0: 40162 lags 5112675666 cycles cpu1: AMD Ryzen 5 3500U with Radeon Vega Mobile Gfx, 2096.04 MHz, 17-18-01 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA,IBPB,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu1: 32KB 64b/line 8-way D-cache, 64KB 64b/line 4-way I-cache, 512KB 64b/line 8-way L2 cache, 4MB 64b/line 16-way L3 cache cpu1: smt 1, core 0, package 0 cpu2 at mainbus0: apid 2 (application processor) tsc: cpu0/cpu2: sync test round 1/2 failed tsc: cpu0/cpu2: cpu0: 18995 lags 5112675645 cycles cpu2: AMD Ryzen 5 3500U with Radeon Vega Mobile Gfx, 2096.04 MHz, 17-18-01 cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA,IBPB,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu2: 32KB 64b/line 8-way D-cache, 64KB 64b/line 4-way I-cache, 512KB 64b/line 8-way L2 cache, 4MB 64b/line 16-way L3 cache cpu2: smt 0, core 1, package 0 cpu3 at mainbus0: apid 3 (application processor) tsc: cpu0/cpu3: sync test round 1/2 failed tsc: cpu0/cpu3: cpu0: 19136 lags 5112675645 cycles cpu3: AMD Ryzen 5 3500U with Radeon Vega Mobile Gfx, 2096.04 MHz, 17-18-01 cpu3:
Re: [v3] amd64: simplify TSC sync testing
Hi, Scotto. I tested your patch on my Ryzen7 box. And I got failed message: tsc: cpu0/cpu1: sync test round 1/2 failed tsc: cpu0/cpu1: cpu1: 1 lags 36 cycles OpenBSD 7.1-current (GENERIC.MP) #3: Wed Jul 6 10:59:06 JST 2022 a...@g2-obsd.my.domain:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 34256752640 (32669MB) avail mem = 33201184768 (31663MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xdb64 (63 entries) bios0: vendor American Megatrends Inc. version "9015" date 03/03/2020 bios0: MouseComputer Co.,Ltd. LM-AG400 acpi0 at bios0: ACPI 6.0 acpi0: sleep states S0 S3 S4 S5 acpi0: tables DSDT FACP APIC FPDT FIDT SSDT MSDM SSDT SSDT MCFG HPET UEFI BGRT TPM2 IVRS PCCT SSDT CRAT CDIT SSDT SSDT SSDT acpi0: wakeup devices GPP0(S4) GPP1(S4) GPP3(S4) GPP4(S4) GPP5(S4) GPP6(S4) GPP7(S4) GPP8(S4) X161(S4) GPP9(S4) X162(S4) GPPA(S4) GPPB(S4) GPPC(S4) GPPD(S4) GPPE(S4) [...] acpitimer0 at acpi0: 3579545 Hz, 32 bits acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: AMD Ryzen 7 3700X 8-Core Processor, 3593.78 MHz, 17-71-00 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,PQM,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,IBPB,STIBP,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 512KB 64b/line 8-way L2 cache, 16MB 64b/line 16-way L3 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges cpu0: apic clock running at 99MHz cpu0: mwait min=64, max=64, C-substates=1.1, IBE cpu1 at mainbus0: apid 2 (application processor) tsc: cpu0/cpu1: sync test round 1/2 failed tsc: cpu0/cpu1: cpu1: 1 lags 36 cycles cpu1: AMD Ryzen 7 3700X 8-Core Processor, 3593.25 MHz, 17-71-00 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,PQM,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,IBPB,STIBP,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu1: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 512KB 64b/line 8-way L2 cache, 16MB 64b/line 16-way L3 cache cpu1: smt 0, core 1, package 0 cpu2 at mainbus0: apid 4 (application processor) tsc: cpu0/cpu2: sync test round 2/2 failed tsc: cpu0/cpu2: cpu2: 1 lags 36 cycles cpu2: AMD Ryzen 7 3700X 8-Core Processor, 3593.25 MHz, 17-71-00 cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,PQM,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,IBPB,STIBP,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu2: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 512KB 64b/line 8-way L2 cache, 16MB 64b/line 16-way L3 cache cpu2: smt 0, core 2, package 0 cpu3 at mainbus0: apid 6 (application processor) cpu3: AMD Ryzen 7 3700X 8-Core Processor, 3593.25 MHz, 17-71-00 cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,PQM,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,IBPB,STIBP,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu3: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 512KB 64b/line 8-way L2 cache, 16MB 64b/line 16-way L3 cache cpu3: smt 0, core 3, package 0 cpu4 at mainbus0: apid 8 (application processor) tsc: cpu0/cpu4: sync test round 1/2 failed tsc: cpu0/cpu4: cpu4: 5 lags 1116 cycles cpu4: AMD Ryzen 7 3700X 8-Core Processor, 3593.26 MHz, 17-71-00 cpu4: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,PQM,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,IBPB,STIBP,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu4: 32KB 64b/line 8-way
Re: [v3] amd64: simplify TSC sync testing
Sorry the `sysctl kern.timecounter` was before your patch. Here's the one after your patch: $ sysctl kern.timecounter kern.timecounter.tick=1 kern.timecounter.timestepwarnings=0 kern.timecounter.hardware=i8254 kern.timecounter.choice=i8254(0) tsc(-1000) acpihpet0(1000) acpitimer0(1000) On Tue, Jul 05, 2022 at 10:31:33PM -0400, Mohamed Aslan wrote: > Hello, > > I just tested your patch on my lenovo e495 laptop, unfortunately > still no tsc. > > $ dmesg | grep 'tsc:' > tsc: cpu0/cpu1 sync round 1: 20468 regressions > tsc: cpu0/cpu1 sync round 1: cpu0 lags cpu1 by 5351060292 cycles > tsc: cpu0/cpu1 sync round 1: cpu1 lags cpu0 by 0 cycles > tsc: cpu0/cpu2 sync round 1: 10272 regressions > tsc: cpu0/cpu2 sync round 1: cpu0 lags cpu2 by 5351060271 cycles > tsc: cpu0/cpu2 sync round 1: cpu2 lags cpu0 by 0 cycles > tsc: cpu0/cpu3 sync round 1: 9709 regressions > tsc: cpu0/cpu3 sync round 1: cpu0 lags cpu3 by 5351060271 cycles > tsc: cpu0/cpu3 sync round 1: cpu3 lags cpu0 by 0 cycles > tsc: cpu0/cpu4 sync round 1: 10386 regressions > tsc: cpu0/cpu4 sync round 1: cpu0 lags cpu4 by 5351060271 cycles > tsc: cpu0/cpu4 sync round 1: cpu4 lags cpu0 by 0 cycles > tsc: cpu0/cpu5 sync round 1: 10408 regressions > tsc: cpu0/cpu5 sync round 1: cpu0 lags cpu5 by 5351060271 cycles > tsc: cpu0/cpu5 sync round 1: cpu5 lags cpu0 by 0 cycles > tsc: cpu0/cpu6 sync round 1: 10102 regressions > tsc: cpu0/cpu6 sync round 1: cpu0 lags cpu6 by 5351060271 cycles > tsc: cpu0/cpu6 sync round 1: cpu6 lags cpu0 by 0 cycles > tsc: cpu0/cpu7 sync round 1: 9336 regressions > tsc: cpu0/cpu7 sync round 1: cpu0 lags cpu7 by 5351060271 cycles > tsc: cpu0/cpu7 sync round 1: cpu7 lags cpu0 by 0 cycles > > $ sysctl kern.timecounter > kern.timecounter.tick=1 > kern.timecounter.timestepwarnings=0 > kern.timecounter.hardware=tsc > kern.timecounter.choice=i8254(0) tsc(2000) acpihpet0(1000) acpitimer0(1000) > > $ sysctl hw > hw.model=AMD Ryzen 5 3500U with Radeon Vega Mobile Gfx > hw.ncpu=8 > hw.vendor=LENOVO > hw.product=20NE0002US > hw.version=ThinkPad E495 > hw.ncpufound=8 > hw.smt=0 > hw.ncpuonline=4 > > > Best, > M. > > > On Mon, Jul 04, 2022 at 09:06:55PM -0500, Scott Cheloha wrote: > > Hi, > > > > Once again, I am trying to change our approach to TSC sync testing to > > eliminate false positive results. Instead of trying to repair the TSC > > by measuring skew, we just spin in a lockless loop looking for skew > > and mark the TSC as broken if we detect any. > > > > This is motivated in part by some multisocket machines that do not use > > the TSC as a timecounter because the current sync test confuses NUMA > > latency for TSC skew. > > > > The only difference between this version and the prior version (v2) is > > that we check whether we have the IA32_TSC_ADJUST register by hand in > > tsc_reset_adjust(). If someone wants to help me rearrange cpu_hatch() > > so we do CPU identification before TSC sync testing we can remove the > > workaround later. > > > > If you have the IA32_TSC_ADJUST register and it is non-zero going into > > the test, you will see something on the console like this: > > > > tsc: cpu5: IA32_TSC_ADJUST: -150 -> 0 > > > > This does *not* mean you failed the test. It just means you probably > > have a bug in your BIOS (or some other firmware) and you should report > > it to your vendor. > > > > If you fail the test you will see something like this: > > > > tsc: cpu0/cpu2: sync test round 1/2 failed > > tsc: cpu0/cpu2: cpu2: 13043 lags 438 cycles > > > > A printout like this would mean that the sync test for cpu2 failed. > > In particular, cpu2's TSC trails cpu0's TSC by at least 438 cycles. > > If this happens for *any* CPU we mark the TSC timecounter as > > defective. > > > > -- > > > > Please test! Send your dmesg, pass or fail. > > > > I am especially interested in: > > > > 1. A test from dv@. Your dual-socket machine has the IA32_TSC_ADJUST > >register but it failed the test running patch v2. Maybe it will pass > >with this version? > > > > 2. Other multisocket machines. > > > > 3. There were reports of TSC issues with OpenBSD VMs running on ESXi. > >What happens when you run with this patch? > > > > 4. OpenBSD VMs on other hypervisors. > > > > 5. Non-Lenovo machines, non-Intel machines. > > > > -Scott > > > > Index: amd64/tsc.c > > === > > RCS file: /cvs/src/sys/arch/amd64/amd64/tsc.c,v > > retrieving revision 1.24 > > diff -u -p -r1.24 tsc.c > > --- amd64/tsc.c 31 Aug 2021 15:11:54 - 1.24 > > +++ amd64/tsc.c 5 Jul 2022 01:52:10 - > > @@ -36,13 +36,6 @@ int tsc_recalibrate; > > uint64_t tsc_frequency; > > inttsc_is_invariant; > > > > -#defineTSC_DRIFT_MAX 250 > > -#define TSC_SKEW_MAX 100 > > -int64_ttsc_drift_observed; > > - > > -volatile int64_t tsc_sync_val; > > -volatile struct cpu_info
Re: [v3] amd64: simplify TSC sync testing
What I meant in my first email is, it seems that before applying your patch, the tsc was used as the hardware counter (no user TSC though), but after applying your patch the i8254 was the one being used. Thanks On Tue, Jul 05, 2022 at 10:34:54PM -0400, Mohamed Aslan wrote: > Sorry the `sysctl kern.timecounter` was before your patch. > > Here's the one after your patch: > $ sysctl kern.timecounter > kern.timecounter.tick=1 > kern.timecounter.timestepwarnings=0 > kern.timecounter.hardware=i8254 > kern.timecounter.choice=i8254(0) tsc(-1000) acpihpet0(1000) acpitimer0(1000) > > > > On Tue, Jul 05, 2022 at 10:31:33PM -0400, Mohamed Aslan wrote: > > Hello, > > > > I just tested your patch on my lenovo e495 laptop, unfortunately > > still no tsc. > > > > $ dmesg | grep 'tsc:' > > tsc: cpu0/cpu1 sync round 1: 20468 regressions > > tsc: cpu0/cpu1 sync round 1: cpu0 lags cpu1 by 5351060292 cycles > > tsc: cpu0/cpu1 sync round 1: cpu1 lags cpu0 by 0 cycles > > tsc: cpu0/cpu2 sync round 1: 10272 regressions > > tsc: cpu0/cpu2 sync round 1: cpu0 lags cpu2 by 5351060271 cycles > > tsc: cpu0/cpu2 sync round 1: cpu2 lags cpu0 by 0 cycles > > tsc: cpu0/cpu3 sync round 1: 9709 regressions > > tsc: cpu0/cpu3 sync round 1: cpu0 lags cpu3 by 5351060271 cycles > > tsc: cpu0/cpu3 sync round 1: cpu3 lags cpu0 by 0 cycles > > tsc: cpu0/cpu4 sync round 1: 10386 regressions > > tsc: cpu0/cpu4 sync round 1: cpu0 lags cpu4 by 5351060271 cycles > > tsc: cpu0/cpu4 sync round 1: cpu4 lags cpu0 by 0 cycles > > tsc: cpu0/cpu5 sync round 1: 10408 regressions > > tsc: cpu0/cpu5 sync round 1: cpu0 lags cpu5 by 5351060271 cycles > > tsc: cpu0/cpu5 sync round 1: cpu5 lags cpu0 by 0 cycles > > tsc: cpu0/cpu6 sync round 1: 10102 regressions > > tsc: cpu0/cpu6 sync round 1: cpu0 lags cpu6 by 5351060271 cycles > > tsc: cpu0/cpu6 sync round 1: cpu6 lags cpu0 by 0 cycles > > tsc: cpu0/cpu7 sync round 1: 9336 regressions > > tsc: cpu0/cpu7 sync round 1: cpu0 lags cpu7 by 5351060271 cycles > > tsc: cpu0/cpu7 sync round 1: cpu7 lags cpu0 by 0 cycles > > > > $ sysctl kern.timecounter > > kern.timecounter.tick=1 > > kern.timecounter.timestepwarnings=0 > > kern.timecounter.hardware=tsc > > kern.timecounter.choice=i8254(0) tsc(2000) acpihpet0(1000) acpitimer0(1000) > > > > $ sysctl hw > > hw.model=AMD Ryzen 5 3500U with Radeon Vega Mobile Gfx > > hw.ncpu=8 > > hw.vendor=LENOVO > > hw.product=20NE0002US > > hw.version=ThinkPad E495 > > hw.ncpufound=8 > > hw.smt=0 > > hw.ncpuonline=4 > > > > > > Best, > > M. > > > > > > On Mon, Jul 04, 2022 at 09:06:55PM -0500, Scott Cheloha wrote: > > > Hi, > > > > > > Once again, I am trying to change our approach to TSC sync testing to > > > eliminate false positive results. Instead of trying to repair the TSC > > > by measuring skew, we just spin in a lockless loop looking for skew > > > and mark the TSC as broken if we detect any. > > > > > > This is motivated in part by some multisocket machines that do not use > > > the TSC as a timecounter because the current sync test confuses NUMA > > > latency for TSC skew. > > > > > > The only difference between this version and the prior version (v2) is > > > that we check whether we have the IA32_TSC_ADJUST register by hand in > > > tsc_reset_adjust(). If someone wants to help me rearrange cpu_hatch() > > > so we do CPU identification before TSC sync testing we can remove the > > > workaround later. > > > > > > If you have the IA32_TSC_ADJUST register and it is non-zero going into > > > the test, you will see something on the console like this: > > > > > > tsc: cpu5: IA32_TSC_ADJUST: -150 -> 0 > > > > > > This does *not* mean you failed the test. It just means you probably > > > have a bug in your BIOS (or some other firmware) and you should report > > > it to your vendor. > > > > > > If you fail the test you will see something like this: > > > > > > tsc: cpu0/cpu2: sync test round 1/2 failed > > > tsc: cpu0/cpu2: cpu2: 13043 lags 438 cycles > > > > > > A printout like this would mean that the sync test for cpu2 failed. > > > In particular, cpu2's TSC trails cpu0's TSC by at least 438 cycles. > > > If this happens for *any* CPU we mark the TSC timecounter as > > > defective. > > > > > > -- > > > > > > Please test! Send your dmesg, pass or fail. > > > > > > I am especially interested in: > > > > > > 1. A test from dv@. Your dual-socket machine has the IA32_TSC_ADJUST > > >register but it failed the test running patch v2. Maybe it will pass > > >with this version? > > > > > > 2. Other multisocket machines. > > > > > > 3. There were reports of TSC issues with OpenBSD VMs running on ESXi. > > >What happens when you run with this patch? > > > > > > 4. OpenBSD VMs on other hypervisors. > > > > > > 5. Non-Lenovo machines, non-Intel machines. > > > > > > -Scott > > > > > > Index: amd64/tsc.c > > > === > > > RCS file: /cvs/src/sys/arch/amd64/amd64/tsc.c,v > > >
Re: [v3] amd64: simplify TSC sync testing
Hello,
I just tested your patch on my lenovo e495 laptop, unfortunately
still no tsc.
$ dmesg | grep 'tsc:'
tsc: cpu0/cpu1 sync round 1: 20468 regressions
tsc: cpu0/cpu1 sync round 1: cpu0 lags cpu1 by 5351060292 cycles
tsc: cpu0/cpu1 sync round 1: cpu1 lags cpu0 by 0 cycles
tsc: cpu0/cpu2 sync round 1: 10272 regressions
tsc: cpu0/cpu2 sync round 1: cpu0 lags cpu2 by 5351060271 cycles
tsc: cpu0/cpu2 sync round 1: cpu2 lags cpu0 by 0 cycles
tsc: cpu0/cpu3 sync round 1: 9709 regressions
tsc: cpu0/cpu3 sync round 1: cpu0 lags cpu3 by 5351060271 cycles
tsc: cpu0/cpu3 sync round 1: cpu3 lags cpu0 by 0 cycles
tsc: cpu0/cpu4 sync round 1: 10386 regressions
tsc: cpu0/cpu4 sync round 1: cpu0 lags cpu4 by 5351060271 cycles
tsc: cpu0/cpu4 sync round 1: cpu4 lags cpu0 by 0 cycles
tsc: cpu0/cpu5 sync round 1: 10408 regressions
tsc: cpu0/cpu5 sync round 1: cpu0 lags cpu5 by 5351060271 cycles
tsc: cpu0/cpu5 sync round 1: cpu5 lags cpu0 by 0 cycles
tsc: cpu0/cpu6 sync round 1: 10102 regressions
tsc: cpu0/cpu6 sync round 1: cpu0 lags cpu6 by 5351060271 cycles
tsc: cpu0/cpu6 sync round 1: cpu6 lags cpu0 by 0 cycles
tsc: cpu0/cpu7 sync round 1: 9336 regressions
tsc: cpu0/cpu7 sync round 1: cpu0 lags cpu7 by 5351060271 cycles
tsc: cpu0/cpu7 sync round 1: cpu7 lags cpu0 by 0 cycles
$ sysctl kern.timecounter
kern.timecounter.tick=1
kern.timecounter.timestepwarnings=0
kern.timecounter.hardware=tsc
kern.timecounter.choice=i8254(0) tsc(2000) acpihpet0(1000) acpitimer0(1000)
$ sysctl hw
hw.model=AMD Ryzen 5 3500U with Radeon Vega Mobile Gfx
hw.ncpu=8
hw.vendor=LENOVO
hw.product=20NE0002US
hw.version=ThinkPad E495
hw.ncpufound=8
hw.smt=0
hw.ncpuonline=4
Best,
M.
On Mon, Jul 04, 2022 at 09:06:55PM -0500, Scott Cheloha wrote:
> Hi,
>
> Once again, I am trying to change our approach to TSC sync testing to
> eliminate false positive results. Instead of trying to repair the TSC
> by measuring skew, we just spin in a lockless loop looking for skew
> and mark the TSC as broken if we detect any.
>
> This is motivated in part by some multisocket machines that do not use
> the TSC as a timecounter because the current sync test confuses NUMA
> latency for TSC skew.
>
> The only difference between this version and the prior version (v2) is
> that we check whether we have the IA32_TSC_ADJUST register by hand in
> tsc_reset_adjust(). If someone wants to help me rearrange cpu_hatch()
> so we do CPU identification before TSC sync testing we can remove the
> workaround later.
>
> If you have the IA32_TSC_ADJUST register and it is non-zero going into
> the test, you will see something on the console like this:
>
> tsc: cpu5: IA32_TSC_ADJUST: -150 -> 0
>
> This does *not* mean you failed the test. It just means you probably
> have a bug in your BIOS (or some other firmware) and you should report
> it to your vendor.
>
> If you fail the test you will see something like this:
>
> tsc: cpu0/cpu2: sync test round 1/2 failed
> tsc: cpu0/cpu2: cpu2: 13043 lags 438 cycles
>
> A printout like this would mean that the sync test for cpu2 failed.
> In particular, cpu2's TSC trails cpu0's TSC by at least 438 cycles.
> If this happens for *any* CPU we mark the TSC timecounter as
> defective.
>
> --
>
> Please test! Send your dmesg, pass or fail.
>
> I am especially interested in:
>
> 1. A test from dv@. Your dual-socket machine has the IA32_TSC_ADJUST
>register but it failed the test running patch v2. Maybe it will pass
>with this version?
>
> 2. Other multisocket machines.
>
> 3. There were reports of TSC issues with OpenBSD VMs running on ESXi.
>What happens when you run with this patch?
>
> 4. OpenBSD VMs on other hypervisors.
>
> 5. Non-Lenovo machines, non-Intel machines.
>
> -Scott
>
> Index: amd64/tsc.c
> ===
> RCS file: /cvs/src/sys/arch/amd64/amd64/tsc.c,v
> retrieving revision 1.24
> diff -u -p -r1.24 tsc.c
> --- amd64/tsc.c 31 Aug 2021 15:11:54 - 1.24
> +++ amd64/tsc.c 5 Jul 2022 01:52:10 -
> @@ -36,13 +36,6 @@ inttsc_recalibrate;
> uint64_t tsc_frequency;
> int tsc_is_invariant;
>
> -#define TSC_DRIFT_MAX 250
> -#define TSC_SKEW_MAX 100
> -int64_t tsc_drift_observed;
> -
> -volatile int64_t tsc_sync_val;
> -volatile struct cpu_info *tsc_sync_cpu;
> -
> u_inttsc_get_timecount(struct timecounter *tc);
> void tsc_delay(int usecs);
>
> @@ -236,22 +229,12 @@ cpu_recalibrate_tsc(struct timecounter *
> u_int
> tsc_get_timecount(struct timecounter *tc)
> {
> - return rdtsc_lfence() + curcpu()->ci_tsc_skew;
> + return rdtsc_lfence();
> }
>
> void
> tsc_timecounter_init(struct cpu_info *ci, uint64_t cpufreq)
> {
> -#ifdef TSC_DEBUG
> - printf("%s: TSC skew=%lld observed drift=%lld\n", ci->ci_dev->dv_xname,
> - (long long)ci->ci_tsc_skew, (long long)tsc_drift_observed);
>
Re: [v3] amd64: simplify TSC sync testing
> On Jul 5, 2022, at 21:31, Mohamed Aslan wrote: > > Hello, > > I just tested your patch on my lenovo e495 laptop, unfortunately > still no tsc. > > $ dmesg | grep 'tsc:' > tsc: cpu0/cpu1 sync round 1: 20468 regressions > tsc: cpu0/cpu1 sync round 1: cpu0 lags cpu1 by 5351060292 cycles > tsc: cpu0/cpu1 sync round 1: cpu1 lags cpu0 by 0 cycles > tsc: cpu0/cpu2 sync round 1: 10272 regressions > tsc: cpu0/cpu2 sync round 1: cpu0 lags cpu2 by 5351060271 cycles > tsc: cpu0/cpu2 sync round 1: cpu2 lags cpu0 by 0 cycles > tsc: cpu0/cpu3 sync round 1: 9709 regressions > tsc: cpu0/cpu3 sync round 1: cpu0 lags cpu3 by 5351060271 cycles > tsc: cpu0/cpu3 sync round 1: cpu3 lags cpu0 by 0 cycles > tsc: cpu0/cpu4 sync round 1: 10386 regressions > tsc: cpu0/cpu4 sync round 1: cpu0 lags cpu4 by 5351060271 cycles > tsc: cpu0/cpu4 sync round 1: cpu4 lags cpu0 by 0 cycles > tsc: cpu0/cpu5 sync round 1: 10408 regressions > tsc: cpu0/cpu5 sync round 1: cpu0 lags cpu5 by 5351060271 cycles > tsc: cpu0/cpu5 sync round 1: cpu5 lags cpu0 by 0 cycles > tsc: cpu0/cpu6 sync round 1: 10102 regressions > tsc: cpu0/cpu6 sync round 1: cpu0 lags cpu6 by 5351060271 cycles > tsc: cpu0/cpu6 sync round 1: cpu6 lags cpu0 by 0 cycles > tsc: cpu0/cpu7 sync round 1: 9336 regressions > tsc: cpu0/cpu7 sync round 1: cpu0 lags cpu7 by 5351060271 cycles > tsc: cpu0/cpu7 sync round 1: cpu7 lags cpu0 by 0 cycles This is not the latest patch. Please apply the latest patch and try again. If possible, please also include your dmesg from a -current kernel with the TSC_DEBUG option set.
Re: [v3] amd64: simplify TSC sync testing
On Wed, Jul 06, 2022 at 09:14:03AM +0900, Yuichiro NAITO wrote: > Hi, Scott. > > I tested your patch on my OpenBSD running on ESXi. > It works fine for me and I never see monotonic clock going backward. > There is nothing extra messages in my dmesg. Great! Thanks for testing.
Re: [v3] amd64: simplify TSC sync testing
On Tue, Jul 05, 2022 at 01:38:32PM -0700, Mike Larkin wrote: > On Mon, Jul 04, 2022 at 09:06:55PM -0500, Scott Cheloha wrote: > > > > [...] > > Here's the output from a 4 socket 80 thread machine. Oh nice. I think this is the biggest machine we've tried so far. > kern.timecounter reports tsc after boot. Excellent. > Looks like this machine doesn't have the adjust MSR? IA32_TSC_ADJUST first appears in the Intel SDM Vol. 3 some time in 2011 or 2012. I can't find the exact revision. (I really wish there was a comprehensive version history for this sort of thing, i.e. this MSR first appeared in the blah-blah uarch, this instruction is available on all uarchs after yada-yada, etc.) There are apparently several versions of the E7-4870 in the E7 "family". If your CPU predates that, or launched 2012-2014, the MSR may not have made the cut. An aside: I cannot find any evidence of AMD supporting this MSR in any processor. It would be really, really nice if they did. If you (or anyone reading) knows anything about this, or whether they have an equivalent MSR, shout it out. > Other than that, machine seems stable. Good, glad to hear it. Thank you for testing. Has this machine had issues using the TSC on -current in the past? (If you have the time) what does the dmesg look like on the -current kernel with TSC_DEBUG enabled?
Re: [v3] amd64: simplify TSC sync testing
Hi, Scott. I tested your patch on my OpenBSD running on ESXi. It works fine for me and I never see monotonic clock going backward. There is nothing extra messages in my dmesg. OpenBSD 7.1-current (GENERIC.MP) #27: Tue Jul 5 14:50:21 JST 2022 yuich...@yuichiro-obsd.soum.co.jp:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 2129526784 (2030MB) avail mem = 2047688704 (1952MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xe0010 (248 entries) bios0: vendor Phoenix Technologies LTD version "6.00" date 12/12/2018 bios0: VMware, Inc. VMware Virtual Platform acpi0 at bios0: ACPI 4.0 acpi0: sleep states S0 S1 S4 S5 acpi0: tables DSDT FACP BOOT APIC MCFG SRAT HPET WAET acpi0: wakeup devices PCI0(S3) USB_(S1) P2P0(S3) S1F0(S3) S2F0(S3) S8F0(S3) S16F(S3) S17F(S3) S18F(S3) S22F(S3) S23F(S3) S24F(S3) S25F(S3) PE40(S3) S1F0(S3) PE50(S3) [...] acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: AMD Ryzen 7 5700G with Radeon Graphics, 3792.28 MHz, 19-50-00 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,PKU,IBPB,XSAVEOPT,XSAVEC,XSAVES cpu0: 32KB 64b/line 8-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 8-way L2 cache cpu0: ITLB 64 4KB entries fully associative, 64 4MB entries fully associative cpu0: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges cpu0: apic clock running at 65MHz cpu1 at mainbus0: apid 2 (application processor) cpu1: AMD Ryzen 7 5700G with Radeon Graphics, 3792.02 MHz, 19-50-00 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,PKU,IBPB,XSAVEOPT,XSAVEC,XSAVES cpu1: 32KB 64b/line 8-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 8-way L2 cache cpu1: ITLB 64 4KB entries fully associative, 64 4MB entries fully associative cpu1: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative cpu1: smt 0, core 0, package 2 cpu2 at mainbus0: apid 4 (application processor) cpu2: AMD Ryzen 7 5700G with Radeon Graphics, 3792.03 MHz, 19-50-00 cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,PKU,IBPB,XSAVEOPT,XSAVEC,XSAVES cpu2: 32KB 64b/line 8-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 8-way L2 cache cpu2: ITLB 64 4KB entries fully associative, 64 4MB entries fully associative cpu2: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative cpu2: smt 0, core 0, package 4 cpu3 at mainbus0: apid 6 (application processor) cpu3: AMD Ryzen 7 5700G with Radeon Graphics, 3792.04 MHz, 19-50-00 cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,PKU,IBPB,XSAVEOPT,XSAVEC,XSAVES cpu3: 32KB 64b/line 8-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 8-way L2 cache cpu3: ITLB 64 4KB entries fully associative, 64 4MB entries fully associative cpu3: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative cpu3: smt 0, core 0, package 6 ioapic0 at mainbus0: apid 1 pa 0xfec0, version 20, 24 pins acpimcfg0 at acpi0 acpimcfg0: addr 0xf000, bus 0-127 acpihpet0 at acpi0: 14318179 Hz acpiprt0 at acpi0: bus 0 (PCI0) acpipci0 at acpi0 PCI0: 0x 0x0011 0x0001 acpicmos0 at acpi0 "PNP0A05" at acpi0 not configured acpiac0 at acpi0: AC unit online acpicpu0 at acpi0: C1(@1 halt!) acpicpu1 at acpi0: C1(@1 halt!) acpicpu2 at acpi0: C1(@1 halt!) acpicpu3 at acpi0: C1(@1 halt!) pvbus0 at mainbus0: VMware vmt0 at pvbus0 pci0 at mainbus0 bus 0 pchb0 at pci0 dev 0 function 0 "Intel 82443BX AGP" rev 0x01 ppb0 at pci0 dev 1 function 0 "Intel 82443BX AGP" rev 0x01 pci1 at ppb0 bus 1 pcib0 at pci0 dev 7 function 0 "Intel 82371AB
Re: [v3] amd64: simplify TSC sync testing
On Mon, Jul 04, 2022 at 09:06:55PM -0500, Scott Cheloha wrote: > Hi, > > Once again, I am trying to change our approach to TSC sync testing to > eliminate false positive results. Instead of trying to repair the TSC > by measuring skew, we just spin in a lockless loop looking for skew > and mark the TSC as broken if we detect any. > > This is motivated in part by some multisocket machines that do not use > the TSC as a timecounter because the current sync test confuses NUMA > latency for TSC skew. > > The only difference between this version and the prior version (v2) is > that we check whether we have the IA32_TSC_ADJUST register by hand in > tsc_reset_adjust(). If someone wants to help me rearrange cpu_hatch() > so we do CPU identification before TSC sync testing we can remove the > workaround later. > > If you have the IA32_TSC_ADJUST register and it is non-zero going into > the test, you will see something on the console like this: > > tsc: cpu5: IA32_TSC_ADJUST: -150 -> 0 > > This does *not* mean you failed the test. It just means you probably > have a bug in your BIOS (or some other firmware) and you should report > it to your vendor. > > If you fail the test you will see something like this: > > tsc: cpu0/cpu2: sync test round 1/2 failed > tsc: cpu0/cpu2: cpu2: 13043 lags 438 cycles > > A printout like this would mean that the sync test for cpu2 failed. > In particular, cpu2's TSC trails cpu0's TSC by at least 438 cycles. > If this happens for *any* CPU we mark the TSC timecounter as > defective. > > -- > > Please test! Send your dmesg, pass or fail. > > I am especially interested in: > > 1. A test from dv@. Your dual-socket machine has the IA32_TSC_ADJUST >register but it failed the test running patch v2. Maybe it will pass >with this version? > > 2. Other multisocket machines. > > 3. There were reports of TSC issues with OpenBSD VMs running on ESXi. >What happens when you run with this patch? > > 4. OpenBSD VMs on other hypervisors. > > 5. Non-Lenovo machines, non-Intel machines. > > -Scott Here's the output from a 4 socket 80 thread machine. kern.timecounter reports tsc after boot. Looks like this machine doesn't have the adjust MSR? Other than that, machine seems stable. -ml OpenBSD 7.1-current (GENERIC.MP) #14: Tue Jul 5 13:19:28 PDT 2022 mlar...@slave.int.azathoth.net:/u/bin/src/OpenBSD/this/src/sys/arch/amd64/compile/GENERIC.MP real mem = 274847727616 (262115MB) avail mem = 266500612096 (254154MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.6 @ 0x7f49c000 (103 entries) bios0: vendor Dell Inc. version "2.11.0" date 06/04/2018 bios0: Dell Inc. PowerEdge R810 acpi0 at bios0: ACPI 3.0 acpi0: sleep states S0 S4 S5 acpi0: tables DSDT FACP APIC SPCR HPET DMAR MCFG WD__ SLIC ERST HEST BERT EINJ SRAT TCPA SSDT acpi0: wakeup devices PCI0(S5) acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Xeon(R) CPU E7- 4870 @ 2.40GHz, 2394.31 MHz, 06-2f-02 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,AES,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,MELTDOWN cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 4-way I-cache, 256KB 64b/line 8-way L2 cache, 30MB 64b/line 24-way L3 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges cpu0: apic clock running at 132MHz cpu0: mwait min=64, max=64, C-substates=0.2.1.1, IBE cpu1 at mainbus0: apid 64 (application processor) cpu1: Intel(R) Xeon(R) CPU E7- 4870 @ 2.40GHz, 2394.02 MHz, 06-2f-02 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,AES,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,MELTDOWN cpu1: 32KB 64b/line 8-way D-cache, 32KB 64b/line 4-way I-cache, 256KB 64b/line 8-way L2 cache, 30MB 64b/line 24-way L3 cache cpu1: smt 0, core 0, package 1 cpu2 at mainbus0: apid 128 (application processor) cpu2: Intel(R) Xeon(R) CPU E7- 4870 @ 2.40GHz, 2394.01 MHz, 06-2f-02 cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,AES,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,MELTDOWN cpu2: 32KB 64b/line 8-way D-cache, 32KB 64b/line 4-way I-cache, 256KB 64b/line 8-way L2 cache, 30MB 64b/line 24-way L3 cache cpu2: smt 0, core 0, package 2 cpu3 at mainbus0: apid 192 (application
Re: [v3] amd64: simplify TSC sync testing
On Tue, Jul 05, 2022 at 06:40:26PM +0200, Stuart Henderson wrote: > On 2022/07/05 11:22, Scott Cheloha wrote: > > On Tue, Jul 05, 2022 at 05:47:51PM +0200, Stuart Henderson wrote: > > > On 2022/07/04 21:06, Scott Cheloha wrote: > > > > 4. OpenBSD VMs on other hypervisors. > > > > > > KVM on proxmox VE 7.1-12 > > > > > > I force acpihpet0 on this; it defaults to pvclock which results in > > > timekeeping so bad that ntpd can't correct > > > > That is an interesting problem. Probably worth looking at pvclock(4) > > separately. > > > > > $ sysctl kern.timecounter > > > kern.timecounter.tick=1 > > > kern.timecounter.timestepwarnings=0 > > > kern.timecounter.hardware=acpihpet0 > > > kern.timecounter.choice=i8254(0) pvclock0(1500) acpihpet0(1000) > > > acpitimer0(1000) > > > > > > OpenBSD 7.1-current (GENERIC.MP) #45: Tue Jul 5 16:11:00 BST 2022 > > > st...@bamboo.spacehopper.org:/sys/arch/amd64/compile/GENERIC.MP > > > real mem = 8573001728 (8175MB) > > > avail mem = 8295833600 (7911MB) > > > random: good seed from bootblocks > > > mpath0 at root > > > scsibus0 at mpath0: 256 targets > > > mainbus0 at root > > > bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf58c0 (10 entries) > > > bios0: vendor SeaBIOS version > > > "rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org" date 04/01/2014 > > > bios0: QEMU Standard PC (i440FX + PIIX, 1996) > > > acpi0 at bios0: ACPI 1.0 > > > acpi0: sleep states S3 S4 S5 > > > acpi0: tables DSDT FACP APIC SSDT HPET WAET > > > acpi0: wakeup devices > > > acpitimer0 at acpi0: 3579545 Hz, 24 bits > > > acpimadt0 at acpi0 addr 0xfee0: PC-AT compat > > > cpu0 at mainbus0: apid 0 (boot processor) > > > cpu0: AMD Ryzen 5 PRO 5650G with Radeon Graphics, 3893.04 MHz, 19-50-00 > > > cpu0: > > > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,CPCTR,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,PKU,IBRS,IBPB,STIBP,SSBD,IBPB,IBRS,STIBP,SSBD,VIRTSSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES > > > > This machine doesn't have the ITSC flag, so we would never consider > > using the TSC as a timecounter. The sync test is not run, but that > > makes sense. > > > > ... is that expected? Should the machine have the ITSC flag? > > > > (I'm not familiar with Proxmox.) > > > > No idea to be honest. The cpu type is set to "host" so it should pass > things through, but perhaps it deliberately filters out ITSC. Mostly > wanted to point it out as a "doesn't make things worse" (and because > you specifically wanted tests on other VMs :) Gotcha, that's okay then.
Re: [v3] amd64: simplify TSC sync testing
On 2022/07/05 11:22, Scott Cheloha wrote: > On Tue, Jul 05, 2022 at 05:47:51PM +0200, Stuart Henderson wrote: > > On 2022/07/04 21:06, Scott Cheloha wrote: > > > 4. OpenBSD VMs on other hypervisors. > > > > KVM on proxmox VE 7.1-12 > > > > I force acpihpet0 on this; it defaults to pvclock which results in > > timekeeping so bad that ntpd can't correct > > That is an interesting problem. Probably worth looking at pvclock(4) > separately. > > > $ sysctl kern.timecounter > > kern.timecounter.tick=1 > > kern.timecounter.timestepwarnings=0 > > kern.timecounter.hardware=acpihpet0 > > kern.timecounter.choice=i8254(0) pvclock0(1500) acpihpet0(1000) > > acpitimer0(1000) > > > > OpenBSD 7.1-current (GENERIC.MP) #45: Tue Jul 5 16:11:00 BST 2022 > > st...@bamboo.spacehopper.org:/sys/arch/amd64/compile/GENERIC.MP > > real mem = 8573001728 (8175MB) > > avail mem = 8295833600 (7911MB) > > random: good seed from bootblocks > > mpath0 at root > > scsibus0 at mpath0: 256 targets > > mainbus0 at root > > bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf58c0 (10 entries) > > bios0: vendor SeaBIOS version > > "rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org" date 04/01/2014 > > bios0: QEMU Standard PC (i440FX + PIIX, 1996) > > acpi0 at bios0: ACPI 1.0 > > acpi0: sleep states S3 S4 S5 > > acpi0: tables DSDT FACP APIC SSDT HPET WAET > > acpi0: wakeup devices > > acpitimer0 at acpi0: 3579545 Hz, 24 bits > > acpimadt0 at acpi0 addr 0xfee0: PC-AT compat > > cpu0 at mainbus0: apid 0 (boot processor) > > cpu0: AMD Ryzen 5 PRO 5650G with Radeon Graphics, 3893.04 MHz, 19-50-00 > > cpu0: > > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,CPCTR,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,PKU,IBRS,IBPB,STIBP,SSBD,IBPB,IBRS,STIBP,SSBD,VIRTSSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES > > This machine doesn't have the ITSC flag, so we would never consider > using the TSC as a timecounter. The sync test is not run, but that > makes sense. > > ... is that expected? Should the machine have the ITSC flag? > > (I'm not familiar with Proxmox.) > No idea to be honest. The cpu type is set to "host" so it should pass things through, but perhaps it deliberately filters out ITSC. Mostly wanted to point it out as a "doesn't make things worse" (and because you specifically wanted tests on other VMs :)
Re: [v3] amd64: simplify TSC sync testing
On Tue, Jul 05, 2022 at 05:47:51PM +0200, Stuart Henderson wrote: > On 2022/07/04 21:06, Scott Cheloha wrote: > > 4. OpenBSD VMs on other hypervisors. > > KVM on proxmox VE 7.1-12 > > I force acpihpet0 on this; it defaults to pvclock which results in > timekeeping so bad that ntpd can't correct That is an interesting problem. Probably worth looking at pvclock(4) separately. > $ sysctl kern.timecounter > kern.timecounter.tick=1 > kern.timecounter.timestepwarnings=0 > kern.timecounter.hardware=acpihpet0 > kern.timecounter.choice=i8254(0) pvclock0(1500) acpihpet0(1000) > acpitimer0(1000) > > OpenBSD 7.1-current (GENERIC.MP) #45: Tue Jul 5 16:11:00 BST 2022 > st...@bamboo.spacehopper.org:/sys/arch/amd64/compile/GENERIC.MP > real mem = 8573001728 (8175MB) > avail mem = 8295833600 (7911MB) > random: good seed from bootblocks > mpath0 at root > scsibus0 at mpath0: 256 targets > mainbus0 at root > bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf58c0 (10 entries) > bios0: vendor SeaBIOS version "rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org" > date 04/01/2014 > bios0: QEMU Standard PC (i440FX + PIIX, 1996) > acpi0 at bios0: ACPI 1.0 > acpi0: sleep states S3 S4 S5 > acpi0: tables DSDT FACP APIC SSDT HPET WAET > acpi0: wakeup devices > acpitimer0 at acpi0: 3579545 Hz, 24 bits > acpimadt0 at acpi0 addr 0xfee0: PC-AT compat > cpu0 at mainbus0: apid 0 (boot processor) > cpu0: AMD Ryzen 5 PRO 5650G with Radeon Graphics, 3893.04 MHz, 19-50-00 > cpu0: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,CPCTR,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,PKU,IBRS,IBPB,STIBP,SSBD,IBPB,IBRS,STIBP,SSBD,VIRTSSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES This machine doesn't have the ITSC flag, so we would never consider using the TSC as a timecounter. The sync test is not run, but that makes sense. ... is that expected? Should the machine have the ITSC flag? (I'm not familiar with Proxmox.)
Re: [v3] amd64: simplify TSC sync testing
On Tue, Jul 05, 2022 at 05:38:04PM +0200, Stuart Henderson wrote: > On 2022/07/04 21:06, Scott Cheloha wrote: > > 2. Other multisocket machines. > > This is from the R620 where I originally discovered the problems > with SMP with the previous TSC test: > > $ dmesg|grep tsc > $ sysctl kern.timecounter > kern.timecounter.tick=1 > kern.timecounter.timestepwarnings=0 > kern.timecounter.hardware=tsc > kern.timecounter.choice=i8254(0) acpihpet0(1000) tsc(2000) acpitimer0(1000) > > --- old Tue Jul 5 15:34:06 2022 > +++ new Tue Jul 5 15:34:08 2022 > @@ -1,7 +1,7 @@ > [snip] Okay, so on the -current kernel the TSC is marked defective, but with this patch (v3) the TSC is fine: you get no printouts on the console from the TSC module. Good, excellent. Thank you for testing again.
Re: [v3] amd64: simplify TSC sync testing
On Tue, Jul 05, 2022 at 10:53:43AM -0400, Dave Voutila wrote: > > Scott Cheloha writes: > > > On Tue, Jul 05, 2022 at 07:15:31AM -0400, Dave Voutila wrote: > >> > >> Scott Cheloha writes: > >> > >> > [...] > >> > > >> > If you fail the test you will see something like this: > >> > > >> > tsc: cpu0/cpu2: sync test round 1/2 failed > >> > tsc: cpu0/cpu2: cpu2: 13043 lags 438 cycles > >> > > >> > A printout like this would mean that the sync test for cpu2 failed. > >> > In particular, cpu2's TSC trails cpu0's TSC by at least 438 cycles. > >> > If this happens for *any* CPU we mark the TSC timecounter as > >> > defective. > >> > >> I think this passes now on my dual-socket Xeon box? > > > > Yes, it passes. The timecounter on your machine should still have a > > quality of 2000, i.e. we didn't mark it defective. > > > >> Full dmesg at the end of the email[1], but just the `tsc:' lines look > >> like: > >> > >> $ grep tsc dmesg.txt > >> tsc: cpu0: IA32_TSC_ADJUST: -5774382067215574 -> 0 > >> tsc: cpu1: IA32_TSC_ADJUST: -5774382076335870 -> 0 > >> tsc: cpu2: IA32_TSC_ADJUST: -5774382073829798 -> 0 > >> tsc: cpu3: IA32_TSC_ADJUST: -5774382071913818 -> 0 > >> tsc: cpu4: IA32_TSC_ADJUST: -5774382075956770 -> 0 > >> tsc: cpu5: IA32_TSC_ADJUST: -5774382074583181 -> 0 > >> tsc: cpu6: IA32_TSC_ADJUST: -5774382073199574 -> 0 > >> tsc: cpu7: IA32_TSC_ADJUST: -5774382076500135 -> 0 > >> tsc: cpu8: IA32_TSC_ADJUST: -5774382074705354 -> 0 > >> tsc: cpu9: IA32_TSC_ADJUST: -5774382075954945 -> 0 > >> tsc: cpu10: IA32_TSC_ADJUST: -5774382070567294 -> 0 > >> tsc: cpu11: IA32_TSC_ADJUST: -5774382075968443 -> 0 > >> tsc: cpu12: IA32_TSC_ADJUST: -5774382067353478 -> 0 > >> tsc: cpu13: IA32_TSC_ADJUST: -5774382071926523 -> 0 > >> tsc: cpu14: IA32_TSC_ADJUST: -5774382074619890 -> 0 > >> tsc: cpu15: IA32_TSC_ADJUST: -5774382070107058 -> 0 > >> tsc: cpu16: IA32_TSC_ADJUST: -5774382076196640 -> 0 > >> tsc: cpu17: IA32_TSC_ADJUST: -5774382075090665 -> 0 > >> tsc: cpu18: IA32_TSC_ADJUST: -5774382073529646 -> 0 > >> tsc: cpu19: IA32_TSC_ADJUST: -5774382076443616 -> 0 > >> tsc: cpu20: IA32_TSC_ADJUST: -5774382074994536 -> 0 > >> tsc: cpu21: IA32_TSC_ADJUST: -5774382076309520 -> 0 > >> tsc: cpu22: IA32_TSC_ADJUST: -5774382070947686 -> 0 > >> tsc: cpu23: IA32_TSC_ADJUST: -5774382073056320 -> 0 > > > > Fascinating. Wonder what the heck it's doing down there. > > > >> It does look like there's a newer BIOS version for this machine, so I'll > >> try updating it later today and repeating the test to see if anything > >> changes. > > After a BIOS update, still similar output. > > "new" bios: > bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xec0f0 (105 entries) > bios0: vendor Dell Inc. version "A34" date 10/19/2020 > bios0: Dell Inc. Precision Tower 7810 > > $ dmesg | grep tsc > tsc: cpu0: IA32_TSC_ADJUST: -4070378216 -> 0 > tsc: cpu1: IA32_TSC_ADJUST: -4081094631 -> 0 > tsc: cpu2: IA32_TSC_ADJUST: -4078853396 -> 0 > tsc: cpu3: IA32_TSC_ADJUST: -4074362824 -> 0 > tsc: cpu4: IA32_TSC_ADJUST: -4080872645 -> 0 > tsc: cpu5: IA32_TSC_ADJUST: -4075673830 -> 0 > tsc: cpu6: IA32_TSC_ADJUST: -4081906959 -> 0 > tsc: cpu7: IA32_TSC_ADJUST: -4073006269 -> 0 > tsc: cpu8: IA32_TSC_ADJUST: -4081803214 -> 0 > tsc: cpu9: IA32_TSC_ADJUST: -4081294540 -> 0 > tsc: cpu10: IA32_TSC_ADJUST: -4079817920 -> 0 > tsc: cpu11: IA32_TSC_ADJUST: -4079871039 -> 0 > tsc: cpu12: IA32_TSC_ADJUST: -4070522580 -> 0 > tsc: cpu13: IA32_TSC_ADJUST: -4077205405 -> 0 > tsc: cpu14: IA32_TSC_ADJUST: -4081797309 -> 0 > tsc: cpu15: IA32_TSC_ADJUST: -4078574630 -> 0 > tsc: cpu16: IA32_TSC_ADJUST: -4081539272 -> 0 > tsc: cpu17: IA32_TSC_ADJUST: -4079657247 -> 0 > tsc: cpu18: IA32_TSC_ADJUST: -4080469326 -> 0 > tsc: cpu19: IA32_TSC_ADJUST: -4073404194 -> 0 > tsc: cpu20: IA32_TSC_ADJUST: -4081473720 -> 0 > tsc: cpu21: IA32_TSC_ADJUST: -4076195877 -> 0 > tsc: cpu22: IA32_TSC_ADJUST: -4077876814 -> 0 > tsc: cpu23: IA32_TSC_ADJUST: -4081863303 -> 0 > > And still a quality tsc :) : > > $ sysctl kern.timecounter > kern.timecounter.tick=1 > kern.timecounter.timestepwarnings=0 > kern.timecounter.hardware=tsc > kern.timecounter.choice=i8254(0) acpihpet0(1000) tsc(2000) acpitimer0(1000) Alrighty, that's "good enough". We "fixed" it! Thank you for testing (again). If this patch is committed your machine will be able to use the TSC as a timecounter without issue. Now, if you have the patience, there is another thing you could do: You could report this to Dell. They will probably bullshit you and refuse to consider what you're saying because you aren't running Windows or even Linux. But it's worth a shot. The problem, in brief, is that the IA32_TSC_ADJUST register on every CPU is non-zero when the kernel boots. And there isn't any reason for them to be non-zero. At least, no reason I can imagine. The spec says the TSCs should all start from zero simultaneously and run at the same fixed frequency. Intentionally desynchronizing them does not serve any obvious purpose. It walks, talks, and
Re: [v3] amd64: simplify TSC sync testing
On 2022/07/04 21:06, Scott Cheloha wrote: > 4. OpenBSD VMs on other hypervisors. KVM on proxmox VE 7.1-12 I force acpihpet0 on this; it defaults to pvclock which results in timekeeping so bad that ntpd can't correct $ sysctl kern.timecounter kern.timecounter.tick=1 kern.timecounter.timestepwarnings=0 kern.timecounter.hardware=acpihpet0 kern.timecounter.choice=i8254(0) pvclock0(1500) acpihpet0(1000) acpitimer0(1000) OpenBSD 7.1-current (GENERIC.MP) #45: Tue Jul 5 16:11:00 BST 2022 st...@bamboo.spacehopper.org:/sys/arch/amd64/compile/GENERIC.MP real mem = 8573001728 (8175MB) avail mem = 8295833600 (7911MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf58c0 (10 entries) bios0: vendor SeaBIOS version "rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org" date 04/01/2014 bios0: QEMU Standard PC (i440FX + PIIX, 1996) acpi0 at bios0: ACPI 1.0 acpi0: sleep states S3 S4 S5 acpi0: tables DSDT FACP APIC SSDT HPET WAET acpi0: wakeup devices acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: AMD Ryzen 5 PRO 5650G with Radeon Graphics, 3893.04 MHz, 19-50-00 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,CPCTR,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,PKU,IBRS,IBPB,STIBP,SSBD,IBPB,IBRS,STIBP,SSBD,VIRTSSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu0: 64KB 64b/line 2-way D-cache, 64KB 64b/line 2-way I-cache cpu0: 512KB 64b/line 16-way L2 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges cpu0: apic clock running at 999MHz cpu1 at mainbus0: apid 1 (application processor) cpu1: AMD Ryzen 5 PRO 5650G with Radeon Graphics, 3892.63 MHz, 19-50-00 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,CPCTR,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,PKU,IBRS,IBPB,STIBP,SSBD,IBPB,IBRS,STIBP,SSBD,VIRTSSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu1: 64KB 64b/line 2-way D-cache, 64KB 64b/line 2-way I-cache cpu1: 512KB 64b/line 16-way L2 cache cpu1: smt 0, core 1, package 0 cpu2 at mainbus0: apid 2 (application processor) cpu2: AMD Ryzen 5 PRO 5650G with Radeon Graphics, 3892.64 MHz, 19-50-00 cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,CPCTR,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,PKU,IBRS,IBPB,STIBP,SSBD,IBPB,IBRS,STIBP,SSBD,VIRTSSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu2: 64KB 64b/line 2-way D-cache, 64KB 64b/line 2-way I-cache cpu2: 512KB 64b/line 16-way L2 cache cpu2: smt 0, core 2, package 0 cpu3 at mainbus0: apid 3 (application processor) cpu3: AMD Ryzen 5 PRO 5650G with Radeon Graphics, 3892.65 MHz, 19-50-00 cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,CPCTR,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,PKU,IBRS,IBPB,STIBP,SSBD,IBPB,IBRS,STIBP,SSBD,VIRTSSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu3: 64KB 64b/line 2-way D-cache, 64KB 64b/line 2-way I-cache cpu3: 512KB 64b/line 16-way L2 cache cpu3: smt 0, core 3, package 0 cpu4 at mainbus0: apid 4 (application processor) cpu4: AMD Ryzen 5 PRO 5650G with Radeon Graphics, 3892.65 MHz, 19-50-00 cpu4: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,CPCTR,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,PKU,IBRS,IBPB,STIBP,SSBD,IBPB,IBRS,STIBP,SSBD,VIRTSSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu4: 64KB 64b/line 2-way D-cache, 64KB 64b/line 2-way I-cache cpu4: 512KB 64b/line 16-way L2 cache cpu4: smt 0, core 4, package 0 cpu5 at mainbus0: apid 5 (application processor) cpu5: AMD Ryzen 5 PRO 5650G with Radeon Graphics, 3892.68 MHz, 19-50-00 cpu5:
Re: [v3] amd64: simplify TSC sync testing
On 2022/07/04 21:06, Scott Cheloha wrote: > 2. Other multisocket machines. This is from the R620 where I originally discovered the problems with SMP with the previous TSC test: $ dmesg|grep tsc $ sysctl kern.timecounter kern.timecounter.tick=1 kern.timecounter.timestepwarnings=0 kern.timecounter.hardware=tsc kern.timecounter.choice=i8254(0) acpihpet0(1000) tsc(2000) acpitimer0(1000) --- old Tue Jul 5 15:34:06 2022 +++ new Tue Jul 5 15:34:08 2022 @@ -1,7 +1,7 @@ -OpenBSD 7.1-current (GENERIC.MP) #582: Mon Jun 13 15:37:01 MDT 2022 -dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP +OpenBSD 7.1-current (GENERIC.MP) #45: Tue Jul 5 16:11:00 BST 2022 +st...@bamboo.spacehopper.org:/sys/arch/amd64/compile/GENERIC.MP real mem = 34295709696 (32706MB) -avail mem = 33238974464 (31699MB) +avail mem = 33238970368 (31699MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets @@ -16,132 +16,127 @@ acpi0: wakeup devices PCI0(S5) PCI1(S5) acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) -cpu0: Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz, 2900.38 MHz, 06-3e-04 +cpu0: Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz, 2900.42 MHz, 06-3e-04 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN -cpu0: 256KB 64b/line 8-way L2 cache +cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB 64b/line 8-way L2 cache, 15MB 64b/line 20-way L3 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges -cpu0: apic clock running at 99MHz +cpu0: apic clock running at 100MHz cpu0: mwait min=64, max=64, C-substates=0.2.1.1, IBE cpu1 at mainbus0: apid 32 (application processor) cpu1: Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz, 1200.01 MHz, 06-3e-04 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN -cpu1: 256KB 64b/line 8-way L2 cache +cpu1: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB 64b/line 8-way L2 cache, 15MB 64b/line 20-way L3 cache cpu1: smt 0, core 0, package 1 cpu2 at mainbus0: apid 2 (application processor) -cpu2: Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz, 2900.01 MHz, 06-3e-04 +cpu2: Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz, 2900.02 MHz, 06-3e-04 cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN -cpu2: 256KB 64b/line 8-way L2 cache +cpu2: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB 64b/line 8-way L2 cache, 15MB 64b/line 20-way L3 cache cpu2: smt 0, core 1, package 0 cpu3 at mainbus0: apid 34 (application processor) -cpu3: Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz, 1200.00 MHz, 06-3e-04 +cpu3: Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz, 1200.01 MHz, 06-3e-04 cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN -cpu3: 256KB 64b/line 8-way L2 cache +cpu3: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB 64b/line 8-way L2 cache, 15MB 64b/line 20-way L3 cache cpu3: smt 0, core 1, package 1 cpu4 at mainbus0: apid 4 (application processor) -cpu4: Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz, 2900.01 MHz, 06-3e-04 +cpu4: Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz, 2900.02 MHz, 06-3e-04 cpu4: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN -cpu4: 256KB 64b/line 8-way L2 cache +cpu4: 32KB 64b/line 8-way D-cache, 32KB
Re: [v3] amd64: simplify TSC sync testing
Scott Cheloha writes: > On Tue, Jul 05, 2022 at 07:15:31AM -0400, Dave Voutila wrote: >> >> Scott Cheloha writes: >> >> > [...] >> > >> > If you fail the test you will see something like this: >> > >> >tsc: cpu0/cpu2: sync test round 1/2 failed >> >tsc: cpu0/cpu2: cpu2: 13043 lags 438 cycles >> > >> > A printout like this would mean that the sync test for cpu2 failed. >> > In particular, cpu2's TSC trails cpu0's TSC by at least 438 cycles. >> > If this happens for *any* CPU we mark the TSC timecounter as >> > defective. >> >> I think this passes now on my dual-socket Xeon box? > > Yes, it passes. The timecounter on your machine should still have a > quality of 2000, i.e. we didn't mark it defective. > >> Full dmesg at the end of the email[1], but just the `tsc:' lines look >> like: >> >> $ grep tsc dmesg.txt >> tsc: cpu0: IA32_TSC_ADJUST: -5774382067215574 -> 0 >> tsc: cpu1: IA32_TSC_ADJUST: -5774382076335870 -> 0 >> tsc: cpu2: IA32_TSC_ADJUST: -5774382073829798 -> 0 >> tsc: cpu3: IA32_TSC_ADJUST: -5774382071913818 -> 0 >> tsc: cpu4: IA32_TSC_ADJUST: -5774382075956770 -> 0 >> tsc: cpu5: IA32_TSC_ADJUST: -5774382074583181 -> 0 >> tsc: cpu6: IA32_TSC_ADJUST: -5774382073199574 -> 0 >> tsc: cpu7: IA32_TSC_ADJUST: -5774382076500135 -> 0 >> tsc: cpu8: IA32_TSC_ADJUST: -5774382074705354 -> 0 >> tsc: cpu9: IA32_TSC_ADJUST: -5774382075954945 -> 0 >> tsc: cpu10: IA32_TSC_ADJUST: -5774382070567294 -> 0 >> tsc: cpu11: IA32_TSC_ADJUST: -5774382075968443 -> 0 >> tsc: cpu12: IA32_TSC_ADJUST: -5774382067353478 -> 0 >> tsc: cpu13: IA32_TSC_ADJUST: -5774382071926523 -> 0 >> tsc: cpu14: IA32_TSC_ADJUST: -5774382074619890 -> 0 >> tsc: cpu15: IA32_TSC_ADJUST: -5774382070107058 -> 0 >> tsc: cpu16: IA32_TSC_ADJUST: -5774382076196640 -> 0 >> tsc: cpu17: IA32_TSC_ADJUST: -5774382075090665 -> 0 >> tsc: cpu18: IA32_TSC_ADJUST: -5774382073529646 -> 0 >> tsc: cpu19: IA32_TSC_ADJUST: -5774382076443616 -> 0 >> tsc: cpu20: IA32_TSC_ADJUST: -5774382074994536 -> 0 >> tsc: cpu21: IA32_TSC_ADJUST: -5774382076309520 -> 0 >> tsc: cpu22: IA32_TSC_ADJUST: -5774382070947686 -> 0 >> tsc: cpu23: IA32_TSC_ADJUST: -5774382073056320 -> 0 > > Fascinating. Wonder what the heck it's doing down there. > >> It does look like there's a newer BIOS version for this machine, so I'll >> try updating it later today and repeating the test to see if anything >> changes. After a BIOS update, still similar output. "new" bios: bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xec0f0 (105 entries) bios0: vendor Dell Inc. version "A34" date 10/19/2020 bios0: Dell Inc. Precision Tower 7810 $ dmesg | grep tsc tsc: cpu0: IA32_TSC_ADJUST: -4070378216 -> 0 tsc: cpu1: IA32_TSC_ADJUST: -4081094631 -> 0 tsc: cpu2: IA32_TSC_ADJUST: -4078853396 -> 0 tsc: cpu3: IA32_TSC_ADJUST: -4074362824 -> 0 tsc: cpu4: IA32_TSC_ADJUST: -4080872645 -> 0 tsc: cpu5: IA32_TSC_ADJUST: -4075673830 -> 0 tsc: cpu6: IA32_TSC_ADJUST: -4081906959 -> 0 tsc: cpu7: IA32_TSC_ADJUST: -4073006269 -> 0 tsc: cpu8: IA32_TSC_ADJUST: -4081803214 -> 0 tsc: cpu9: IA32_TSC_ADJUST: -4081294540 -> 0 tsc: cpu10: IA32_TSC_ADJUST: -4079817920 -> 0 tsc: cpu11: IA32_TSC_ADJUST: -4079871039 -> 0 tsc: cpu12: IA32_TSC_ADJUST: -4070522580 -> 0 tsc: cpu13: IA32_TSC_ADJUST: -4077205405 -> 0 tsc: cpu14: IA32_TSC_ADJUST: -4081797309 -> 0 tsc: cpu15: IA32_TSC_ADJUST: -4078574630 -> 0 tsc: cpu16: IA32_TSC_ADJUST: -4081539272 -> 0 tsc: cpu17: IA32_TSC_ADJUST: -4079657247 -> 0 tsc: cpu18: IA32_TSC_ADJUST: -4080469326 -> 0 tsc: cpu19: IA32_TSC_ADJUST: -4073404194 -> 0 tsc: cpu20: IA32_TSC_ADJUST: -4081473720 -> 0 tsc: cpu21: IA32_TSC_ADJUST: -4076195877 -> 0 tsc: cpu22: IA32_TSC_ADJUST: -4077876814 -> 0 tsc: cpu23: IA32_TSC_ADJUST: -4081863303 -> 0 And still a quality tsc :) : $ sysctl kern.timecounter kern.timecounter.tick=1 kern.timecounter.timestepwarnings=0 kern.timecounter.hardware=tsc kern.timecounter.choice=i8254(0) acpihpet0(1000) tsc(2000) acpitimer0(1000) -dv
Re: [v3] amd64: simplify TSC sync testing
On Tue, Jul 05, 2022 at 07:15:31AM -0400, Dave Voutila wrote: > > Scott Cheloha writes: > > > [...] > > > > If you fail the test you will see something like this: > > > > tsc: cpu0/cpu2: sync test round 1/2 failed > > tsc: cpu0/cpu2: cpu2: 13043 lags 438 cycles > > > > A printout like this would mean that the sync test for cpu2 failed. > > In particular, cpu2's TSC trails cpu0's TSC by at least 438 cycles. > > If this happens for *any* CPU we mark the TSC timecounter as > > defective. > > I think this passes now on my dual-socket Xeon box? Yes, it passes. The timecounter on your machine should still have a quality of 2000, i.e. we didn't mark it defective. > Full dmesg at the end of the email[1], but just the `tsc:' lines look > like: > > $ grep tsc dmesg.txt > tsc: cpu0: IA32_TSC_ADJUST: -5774382067215574 -> 0 > tsc: cpu1: IA32_TSC_ADJUST: -5774382076335870 -> 0 > tsc: cpu2: IA32_TSC_ADJUST: -5774382073829798 -> 0 > tsc: cpu3: IA32_TSC_ADJUST: -5774382071913818 -> 0 > tsc: cpu4: IA32_TSC_ADJUST: -5774382075956770 -> 0 > tsc: cpu5: IA32_TSC_ADJUST: -5774382074583181 -> 0 > tsc: cpu6: IA32_TSC_ADJUST: -5774382073199574 -> 0 > tsc: cpu7: IA32_TSC_ADJUST: -5774382076500135 -> 0 > tsc: cpu8: IA32_TSC_ADJUST: -5774382074705354 -> 0 > tsc: cpu9: IA32_TSC_ADJUST: -5774382075954945 -> 0 > tsc: cpu10: IA32_TSC_ADJUST: -5774382070567294 -> 0 > tsc: cpu11: IA32_TSC_ADJUST: -5774382075968443 -> 0 > tsc: cpu12: IA32_TSC_ADJUST: -5774382067353478 -> 0 > tsc: cpu13: IA32_TSC_ADJUST: -5774382071926523 -> 0 > tsc: cpu14: IA32_TSC_ADJUST: -5774382074619890 -> 0 > tsc: cpu15: IA32_TSC_ADJUST: -5774382070107058 -> 0 > tsc: cpu16: IA32_TSC_ADJUST: -5774382076196640 -> 0 > tsc: cpu17: IA32_TSC_ADJUST: -5774382075090665 -> 0 > tsc: cpu18: IA32_TSC_ADJUST: -5774382073529646 -> 0 > tsc: cpu19: IA32_TSC_ADJUST: -5774382076443616 -> 0 > tsc: cpu20: IA32_TSC_ADJUST: -5774382074994536 -> 0 > tsc: cpu21: IA32_TSC_ADJUST: -5774382076309520 -> 0 > tsc: cpu22: IA32_TSC_ADJUST: -5774382070947686 -> 0 > tsc: cpu23: IA32_TSC_ADJUST: -5774382073056320 -> 0 Fascinating. Wonder what the heck it's doing down there. > It does look like there's a newer BIOS version for this machine, so I'll > try updating it later today and repeating the test to see if anything > changes. Sure thing, thanks for testing.
Re: [v3] amd64: simplify TSC sync testing
Scott Cheloha writes: > Hi, > > Once again, I am trying to change our approach to TSC sync testing to > eliminate false positive results. Instead of trying to repair the TSC > by measuring skew, we just spin in a lockless loop looking for skew > and mark the TSC as broken if we detect any. > > This is motivated in part by some multisocket machines that do not use > the TSC as a timecounter because the current sync test confuses NUMA > latency for TSC skew. > > The only difference between this version and the prior version (v2) is > that we check whether we have the IA32_TSC_ADJUST register by hand in > tsc_reset_adjust(). If someone wants to help me rearrange cpu_hatch() > so we do CPU identification before TSC sync testing we can remove the > workaround later. > > If you have the IA32_TSC_ADJUST register and it is non-zero going into > the test, you will see something on the console like this: > > tsc: cpu5: IA32_TSC_ADJUST: -150 -> 0 > > This does *not* mean you failed the test. It just means you probably > have a bug in your BIOS (or some other firmware) and you should report > it to your vendor. > > If you fail the test you will see something like this: > > tsc: cpu0/cpu2: sync test round 1/2 failed > tsc: cpu0/cpu2: cpu2: 13043 lags 438 cycles > > A printout like this would mean that the sync test for cpu2 failed. > In particular, cpu2's TSC trails cpu0's TSC by at least 438 cycles. > If this happens for *any* CPU we mark the TSC timecounter as > defective. I think this passes now on my dual-socket Xeon box? Full dmesg at the end of the email[1], but just the `tsc:' lines look like: $ grep tsc dmesg.txt tsc: cpu0: IA32_TSC_ADJUST: -5774382067215574 -> 0 tsc: cpu1: IA32_TSC_ADJUST: -5774382076335870 -> 0 tsc: cpu2: IA32_TSC_ADJUST: -5774382073829798 -> 0 tsc: cpu3: IA32_TSC_ADJUST: -5774382071913818 -> 0 tsc: cpu4: IA32_TSC_ADJUST: -5774382075956770 -> 0 tsc: cpu5: IA32_TSC_ADJUST: -5774382074583181 -> 0 tsc: cpu6: IA32_TSC_ADJUST: -5774382073199574 -> 0 tsc: cpu7: IA32_TSC_ADJUST: -5774382076500135 -> 0 tsc: cpu8: IA32_TSC_ADJUST: -5774382074705354 -> 0 tsc: cpu9: IA32_TSC_ADJUST: -5774382075954945 -> 0 tsc: cpu10: IA32_TSC_ADJUST: -5774382070567294 -> 0 tsc: cpu11: IA32_TSC_ADJUST: -5774382075968443 -> 0 tsc: cpu12: IA32_TSC_ADJUST: -5774382067353478 -> 0 tsc: cpu13: IA32_TSC_ADJUST: -5774382071926523 -> 0 tsc: cpu14: IA32_TSC_ADJUST: -5774382074619890 -> 0 tsc: cpu15: IA32_TSC_ADJUST: -5774382070107058 -> 0 tsc: cpu16: IA32_TSC_ADJUST: -5774382076196640 -> 0 tsc: cpu17: IA32_TSC_ADJUST: -5774382075090665 -> 0 tsc: cpu18: IA32_TSC_ADJUST: -5774382073529646 -> 0 tsc: cpu19: IA32_TSC_ADJUST: -5774382076443616 -> 0 tsc: cpu20: IA32_TSC_ADJUST: -5774382074994536 -> 0 tsc: cpu21: IA32_TSC_ADJUST: -5774382076309520 -> 0 tsc: cpu22: IA32_TSC_ADJUST: -5774382070947686 -> 0 tsc: cpu23: IA32_TSC_ADJUST: -5774382073056320 -> 0 It does look like there's a newer BIOS version for this machine, so I'll try updating it later today and repeating the test to see if anything changes. -dv [1] OpenBSD 7.1-current (GENERIC.MP) #6: Tue Jul 5 07:07:34 EDT 2022 d...@minmin.sisu.home:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 85769121792 (81795MB) avail mem = 83152367616 (79300MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xec0f0 (105 entries) bios0: vendor Dell Inc. version "A32" date 09/25/2019 bios0: Dell Inc. Precision Tower 7810 acpi0 at bios0: ACPI 5.0 acpi0: sleep states S0 S3 S4 S5 acpi0: tables DSDT FACP APIC FPDT FIDT MCFG UEFI HPET MSCT SLIT SRAT SRAT WDDT SSDT NITR SLIC MSDM DMAR ASF! acpi0: wakeup devices IP2P(S3) RP01(S4) RP02(S4) RP03(S4) RP04(S4) RP06(S4) RP07(S4) RP08(S4) BR1A(S4) BR1B(S4) BR2A(S4) BR2B(S4) BR2C(S4) BR2D(S4) BR3A(S4) BR3B(S4) [...] acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Xeon(R) CPU E5-2643 v3 @ 3.40GHz, 3392.65 MHz, 06-3f-02 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB 64b/line 8-way L2 cache, 20MB 64b/line 20-way L3 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges cpu0: apic clock running at 99MHz cpu0: mwait min=64, max=64, C-substates=0.2.1.2, IBE cpu1 at mainbus0: apid 2 (application processor) tsc: cpu0: IA32_TSC_ADJUST: -5774382067215574 -> 0 tsc: cpu1: IA32_TSC_ADJUST: -5774382076335870 -> 0 cpu1: Intel(R) Xeon(R) CPU
