FreeBSD on AMD Epyc boards

[Mike Tancsa]

To have a bit of a work around for the Intel Meltdown bug (yes, no
Spectre), I wanted to try out some AMD based CPUs.  So far so good using
a SuperMicro H11SSL-i.  A decent server board using an Epyc CPU.  All
the things you need and expect for a server grade MB

ipmi to provide remote management (SoL to BIOS and OS) and hardware
info.  The ipmi.ko driver works great with RELENG11.  This allows for
hardware watchdog support too.

amdtemp from CURRENT also works to provide cpu temp info, although
ipmitool does this as well.

Seems to sit at about 50W when idle and tops out at about 180W with 2
SSD drives and all cores busy.


Note, the bug in
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=225584
and discussed in
https://reviews.freebsd.org/D14347
manifests itself fairly readily. However, the patch fixes the problem

Attached is some dmesg info for the curious.  Seems like a decent board
for FreeBSD and we are going to start deploying in a couple of spots
once we do some more burn in and testing.

---Mike





-- 
---
Mike Tancsa, tel +1 519 651 3400 x203
Sentex Communications, m...@sentex.net
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada
Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.1-STABLE #0 r329241M: Tue Feb 13 16:20:58 EST 2018
mdtan...@epyc-bsd.sentex.ca:/usr/obj/usr/src/sys/server amd64
FreeBSD clang version 5.0.1 (tags/RELEASE_501/final 320880) (based on LLVM 
5.0.1)
SRAT: No memory found for CPU 0
VT(vga): resolution 640x480
CPU: AMD EPYC 7281 16-Core Processor (2100.05-MHz K8-class CPU)
  Origin="AuthenticAMD"  Id=0x800f12  Family=0x17  Model=0x1  Stepping=2
  
Features=0x178bfbff
  
Features2=0x7ed8320b
  AMD Features=0x2e500800
  AMD 
Features2=0x35c233ff
  Structured Extended 
Features=0x209c01a9
  XSAVE Features=0xf
  AMD Extended Feature Extensions ID EBX=0x7
  SVM: NP,NRIP,VClean,AFlush,DAssist,NAsids=32768
  TSC: P-state invariant, performance statistics
real memory  = 34359738368 (32768 MB)
avail memory = 33215496192 (31676 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table: < >
FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs
FreeBSD/SMP: 1 package(s) x 16 core(s) x 2 hardware threads
random: unblocking device.
ioapic0: Changing APIC ID to 128
ioapic1: Changing APIC ID to 129
ioapic2: Changing APIC ID to 130
ioapic3: Changing APIC ID to 131
ioapic4: Changing APIC ID to 132
ioapic0  irqs 0-23 on motherboard
ioapic1  irqs 24-55 on motherboard
ioapic2  irqs 56-87 on motherboard
ioapic3  irqs 88-119 on motherboard
ioapic4  irqs 120-151 on motherboard
SMP: AP CPU #27 Launched!
SMP: AP CPU #19 Launched!
SMP: AP CPU #11 Launched!
SMP: AP CPU #9 Launched!
SMP: AP CPU #13 Launched!
SMP: AP CPU #29 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #26 Launched!
SMP: AP CPU #28 Launched!
SMP: AP CPU #31 Launched!
SMP: AP CPU #8 Launched!
SMP: AP CPU #23 Launched!
SMP: AP CPU #21 Launched!
SMP: AP CPU #22 Launched!
SMP: AP CPU #20 Launched!
SMP: AP CPU #17 Launched!
SMP: AP CPU #18 Launched!
SMP: AP CPU #16 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #30 Launched!
SMP: AP CPU #10 Launched!
SMP: AP CPU #12 Launched!
SMP: AP CPU #25 Launched!
SMP: AP CPU #24 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #15 Launched!
SMP: AP CPU #14 Launched!
Timecounter "TSC" frequency 2100049917 Hz quality 1000
random: entropy device external interface
netmap: loaded module
module_register_init: MOD_LOAD (vesa, 0x80cf13f0, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
kbd1 at kbdmux0
nexus0
vtvga0:  on motherboard
cryptosoft0:  on motherboard
aesni0:  on motherboard
acpi0:  on motherboard
acpi0: Power Button (fixed)
cpu0:  on acpi0
cpu1:  on acpi0
cpu2:  on acpi0
cpu3:  on acpi0
cpu4:  on acpi0
cpu5:  on acpi0
cpu6:  on acpi0
cpu7:  on acpi0
cpu8:  on acpi0
cpu9:  on acpi0
cpu10:  on acpi0
cpu11:  on acpi0
cpu12:  on acpi0
cpu13:  on acpi0
cpu14:  on acpi0
cpu15:  on acpi0
cpu16:  on acpi0
cpu17:  on acpi0
cpu18:  on acpi0
cpu19:  on acpi0
cpu20:  on acpi0
cpu21:  on acpi0
cpu22:  on acpi0
cpu23:  on acpi0
cpu24:  on acpi0
cpu25:  on acpi0
cpu26:  on acpi0

Re: Ryzen issues on FreeBSD ? (with sort of workaround)

[Eric van Gyzen]

On 02/12/2018 21:54, Peter Moody wrote:
>> I'm having really good luck with the kernel patch attached to this
>> message:
>> https://docs.freebsd.org/cgi/getmsg.cgi?fetch=417183+0+archive/2018/freebsd-hackers/20180211.freebsd-hackers
> 
> I'm new to this; what are the chances that this gets into -STABLE in
> the near future?

Pretty high, I imagine.  Add yourself to these to stay informed:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=225584

https://reviews.freebsd.org/D14347

Eric
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Ryzen issues on FreeBSD ? (summary of 4 issues) (seemingly solved!)

[Mike Tancsa]

OK, this is all mostly solved for me it seems.

points below inline


On 1/24/2018 9:42 AM, Mike Tancsa wrote:
> I think perhaps a good time to summarize as a few issues seem to be going on
> 
> a) fragile BIOS settings. There seems to be a number of issues around
> RAM speeds and disabled C-STATES that impact stability.  Specifically,
> lowering the default frequency from 2400 to 2133 seems to help some
> users with crashes / lockups under heavy loads.

Also disabling core boost on non X cpus (ie 1600 vs 1600x) and making
sure the CPU is not overheating.  On my ASUS board using a back ported
version of amdtemp and amdsmn I confirmed the temp does not go above 50C
at full load.  Setting the FAN speed to turbo seems to help reduce the
max temp the CPU would get.

> b) CPUs manufactured prior to week 25 (some say week 33?) have a
> hardware defect that manifests itself as segfaults in heavy compiles.  I
> was able to confirm this on 1 of the CPUs I had using a Linux setup. It
> seems to confirm this, you need to physically look at the CPU for the
> manufacturing date :( Not sure how to trigger it on FreeBSD reliably,
> but there is a github project I used to verify on Linux
> (https://github.com/suaefar/ryzen-test)

AMD sent me 3 new CPUs without issue.  Turn around was about 1 week from
Canada to the US and back.

> 
> c) The idle lockup bug.  This *seems* to be confirmed on Linux as well
> http://blog.programster.org/ubuntu-16-04-compile-custom-kernel-for-ryzen
> and
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1690085

Perhaps the settings in a), as well as the most recent BIOS update seems
to have fixed this issue for me.  It sure seemed like a hardware issue,
but then again it could be a side effect of d). However, I was never
able to break into the debugger using a debugging kernel in HEAD so I
suspect it was more hardware related than anything.

BIOS Information
Vendor: American Megatrends Inc.
Version: 3803
Release Date: 01/22/2018
Address: 0xF
This is on a
Product Name: PRIME X370-PRO
Version: Rev X.0x


> 
> d) Compile failures of some ports.  For myself and one other user,
> compiling net/samba47 reliably hangs in roughly the same place.  Its not
> clear if this is related to any of the above bugs or not.

This too seems to be fixed!
The patch in
https://docs.freebsd.org/cgi/getmsg.cgi?fetch=417183+0+archive/2018/freebsd-hackers/20180211.freebsd-hackers

seems to stop the deadlock. I did 90 builds on RELENG_11 with this patch
over night and no deadlocks. For half the builds I had 2 guest VMs also
building. For the second half, it was the only thing running on the box
and its working as expected

All this just in time for my Epyc based system to arrive!


---Mike
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"