Re: reboot loop on -current, one machine of several

[Nick Holland]

On 11/13/17 14:24, Gregory Edigarov wrote:
...
>>   scsibus1 at ahci0: 32 targets
>> -sd0 at scsibus1 targ 2 lun 0:  SCSI3 0/direct 
>> fixed naa.50025388400562d4
>> +sd0 at scsibus1 targ 0 lun 0:  SCSI3 0/direct 
>> fixed naa.50025388400563fe
>>   sd0: 976762MB, 512 bytes/sector, 2000409264 sectors, thin
>> -sd1 at scsibus1 targ 3 lun 0:  SCSI3 0/direct 
>> fixed naa.5002538c70007b02
>> -sd1: 1953514MB, 512 bytes/sector, 4000797360 sectors, thin
>> +cd0 at scsibus1 targ 1 lun 0:  ATAPI 5/cdrom 
>> removable
>>   ichiic0 at pci0 dev 31 function 3 "Intel 6 Series SMBus" rev 0x04: apic 0 
>> int 19
>>   iic0 at ichiic0

> My suspicion goes to SSDs. one of them have somehow become bad.

I'm not able to say "no" to that.  Been kinda leaning that direction,
myself.
These have been troublesome little beasts.  Got several of the Samsung
850 series in this project, and never had so many problems with storage
since I tried some off-brand (JTI?) disks around 20 years ago.  Yes, I
know, lots of people think these are the best around (Samsung, not JTI).
 *shrug*

However, I did do a dd read over the first few GB (entire 'a' partition,
partition table, mbr, etc.) of the disk to see if there were any read
errors -- none.  Whatever that's worth.

If all else fails, I'll be moving the function to spare hw and totally
rebuild this machine and see if it fixes it.

Nick.

Re: reboot loop on -current, one machine of several

[Gregory Edigarov]

On 12.11.17 21:59, Nick Holland wrote:

On 11/12/17 14:13, Otto Moerbeek wrote:

On Sun, Nov 12, 2017 at 01:28:39PM -0500, Nick Holland wrote:


Help.

I was upgrading a few very similar machines to -current today.
ONE of the three decided to be unpleasant.  The thing has a
serial console, and but it is about 370km from me. :-/

Upgrade from Sep 9 current to today's current via bsd.rd, just
like the other two.

Upon reboot, it does this (from /boot) :

booting hd0a:/bsd: 8484712+2429968+244048+0+667648 [636809heap full 
(0x9d304+65536)

And then reboots the system, as if from power-down/power-up.
(already something I haven't seen before)

Reboot from "bsd.rd" and "bsd.sp", same results.  reboot from "obsd"
(Sept 9), same results.  Not a kernel problem, it seems.  About this
point, I'm starting to think how the serial console has let me down.

I remember how to bring up a DRAC remote CD image via ssh tunnels
to the drac and how to run java in a windows browser, and
reboot off the remote CD image, do another upgrade, all goes fine
(again), but upon reboot, same results...  "heap full" and reboot.

Boot from remote CD, at the boot> prompt, enter "boot hd0a:/bsd",
and it boots Just Fine from the local hard disk (only boot pulled
from the remote CD).  Boot loader!  Reinstalled boot:

# installboot -v sd0
Using / as root
installing bootstrap on /dev/rsd0c
using first-stage /usr/mdec/biosboot, second-stage /usr/mdec/boot
copying /usr/mdec/boot to /boot
/boot is 3 blocks x 32768 bytes
fs block shift 3; part offset 64; inode block 24, offset 2088
master boot record (MBR) at sector 0
 partition 3: type 0xA6 offset 64 size 2000397671
/usr/mdec/biosboot will be written at sector 64

good, right?

Reboot off local hard disk, boom.  problem is still there.  maybe
not the boot loader. :-/

Verified /boot on trouble system and good system are the same.

I'm not going to cry "bug", since there are two nearly identical
systems working just fine.  But I can't think of what I did wrong
or what to do to fix it.

Suggestions?

You are hitting -DHEAP_LIMIT=0xA in /boot. The code is in libsa/alloa.c

No idea why. But something in that system is different.

You do have one weird line in your disklabel output: a filesystem
mounted on swap?

that's an mfs.  This application has one directory which has a HUGE
benefit to an MFS for tmp files.  Though the reboot happens long before
the mfs is created.


  scsibus1 at ahci0: 32 targets
-sd0 at scsibus1 targ 2 lun 0:  SCSI3 0/direct 
fixed naa.50025388400562d4
+sd0 at scsibus1 targ 0 lun 0:  SCSI3 0/direct 
fixed naa.50025388400563fe
  sd0: 976762MB, 512 bytes/sector, 2000409264 sectors, thin
-sd1 at scsibus1 targ 3 lun 0:  SCSI3 0/direct 
fixed naa.5002538c70007b02
-sd1: 1953514MB, 512 bytes/sector, 4000797360 sectors, thin
+cd0 at scsibus1 targ 1 lun 0:  ATAPI 5/cdrom 
removable
  ichiic0 at pci0 dev 31 function 3 "Intel 6 Series SMBus" rev 0x04: apic 0 int 
19
  iic0 at ichiic0

My suspicion goes to SSDs. one of them have somehow become bad.


Nick.

Re: reboot loop on -current, one machine of several

[Mihai Popescu]

> booting hd0a:/bsd: 8484712+2429968+244048+0+667648 [636809heap full 
> (0x9d304+65536)

Maybe a corrupted or too big /bsd file?

I am curious about the cause.

Re: reboot loop on -current, one machine of several

[Nick Holland]

On 11/12/17 14:13, Otto Moerbeek wrote:
> On Sun, Nov 12, 2017 at 01:28:39PM -0500, Nick Holland wrote:
> 
>> Help.
>> 
>> I was upgrading a few very similar machines to -current today.
>> ONE of the three decided to be unpleasant.  The thing has a
>> serial console, and but it is about 370km from me. :-/
>> 
>> Upgrade from Sep 9 current to today's current via bsd.rd, just
>> like the other two.
>> 
>> Upon reboot, it does this (from /boot) :
>> 
>> booting hd0a:/bsd: 8484712+2429968+244048+0+667648 [636809heap full 
>> (0x9d304+65536)
>> 
>> And then reboots the system, as if from power-down/power-up.
>> (already something I haven't seen before)
>> 
>> Reboot from "bsd.rd" and "bsd.sp", same results.  reboot from "obsd"
>> (Sept 9), same results.  Not a kernel problem, it seems.  About this
>> point, I'm starting to think how the serial console has let me down.
>> 
>> I remember how to bring up a DRAC remote CD image via ssh tunnels
>> to the drac and how to run java in a windows browser, and
>> reboot off the remote CD image, do another upgrade, all goes fine
>> (again), but upon reboot, same results...  "heap full" and reboot.
>> 
>> Boot from remote CD, at the boot> prompt, enter "boot hd0a:/bsd",
>> and it boots Just Fine from the local hard disk (only boot pulled
>> from the remote CD).  Boot loader!  Reinstalled boot:
>> 
>> # installboot -v sd0
>> Using / as root
>> installing bootstrap on /dev/rsd0c
>> using first-stage /usr/mdec/biosboot, second-stage /usr/mdec/boot
>> copying /usr/mdec/boot to /boot
>> /boot is 3 blocks x 32768 bytes
>> fs block shift 3; part offset 64; inode block 24, offset 2088
>> master boot record (MBR) at sector 0
>> partition 3: type 0xA6 offset 64 size 2000397671
>> /usr/mdec/biosboot will be written at sector 64
>> 
>> good, right?
>> 
>> Reboot off local hard disk, boom.  problem is still there.  maybe
>> not the boot loader. :-/
>> 
>> Verified /boot on trouble system and good system are the same.  
>> 
>> I'm not going to cry "bug", since there are two nearly identical
>> systems working just fine.  But I can't think of what I did wrong
>> or what to do to fix it.
>> 
>> Suggestions?
> 
> You are hitting -DHEAP_LIMIT=0xA in /boot. The code is in libsa/alloa.c
> 
> No idea why. But something in that system is different. 
> 
> You do have one weird line in your disklabel output: a filesystem
> mounted on swap? 

that's an mfs.  This application has one directory which has a HUGE
benefit to an MFS for tmp files.  Though the reboot happens long before
the mfs is created.

$ more /etc/fstab   

cde728ba2c9bbe7.b none swap sw
ccde728ba2c9bbe7.a / ffs rw,noatime 1 1
ccde728ba2c9bbe7.h /home ffs rw,noatime,nodev,nosuid 1 2
ccde728ba2c9bbe7.e /tmp ffs rw,noatime,nodev,nosuid 1 2
ccde728ba2c9bbe7.d /usr ffs rw,noatime,nodev 1 2
ccde728ba2c9bbe7.f /var ffs rw,noatime,nodev,nosuid 1 2
ccde728ba2c9bbe7.g /repo ffs rw,noatime,nodev 1 2
ccde728ba2c9bbe7.i /repo/anoncvs/dev ffs rw,noatime,nosuid 1 2
/dev/sd0b /repo/anoncvs/tmp mfs rw,nodev,nosuid,-m=1,-s=3072000,-i=2048 0 0

> Can you boot into single user mode?

nope.  Considering how fast the reboot happens, I wouldn't have expected
it to, unless something is very different very early in the boot process.
This is what happened:

On the console:
Using drive 0, partion 3.
Loading...
probing: pc0 com0 com1 mem[631K 3038M 2M 68K 72K 176k 64K 13312M a20=on]
disk: fd0 hd0+
>> OpenBSD/amd64 BOOT 3.33
switching console to com0

and then on the serial console:
>> OpenBSD/amd64 BOOT 3.33  
boot> boot -s   
booting hd0a:/bsd: 8484304+2429960+244080+0+667648 [643739heap full 
(0x9d4fc+65536)

(boom. reboot)

here's a dmesg diff between the "good" and "bad" machines...
$ diff -u dmesg.good dmesg.bad   
--- dmesg.good  Sun Nov 12 14:51:30 2017
+++ dmesg.bad   Sun Nov 12 14:51:21 2017
@@ -1,7 +1,7 @@
 OpenBSD 6.2-current (GENERIC.MP) #203: Sat Nov 11 19:01:19 MST 2017
 dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
 real mem = 17131339776 (16337MB)
-avail mem = 16605302784 (15836MB)
+avail mem = 16605294592 (15836MB)
 mpath0 at root
 scsibus0 at mpath0: 256 targets
 mainbus0 at root
@@ -16,46 +16,46 @@
 acpihpet0 at acpi0: 14318179 Hz
 acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
 cpu0 at mainbus0: apid 0 (boot processor)
-cpu0: Intel(R) Xeon(R) CPU E31230 @ 3.20GHz, 3193.18 MHz
+cpu0: Intel(R) Xeon(R) CPU E31230 @ 3.20GHz, 3193.22 MHz
 cpu0: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,ARAT
 cpu0: 256KB 64b/line 8-way L2 cache
-acpihpet0: recalibrated 

Re: reboot loop on -current, one machine of several

[Otto Moerbeek]

On Sun, Nov 12, 2017 at 01:28:39PM -0500, Nick Holland wrote:

> Help.
> 
> I was upgrading a few very similar machines to -current today.
> ONE of the three decided to be unpleasant.  The thing has a
> serial console, and but it is about 370km from me. :-/
> 
> Upgrade from Sep 9 current to today's current via bsd.rd, just
> like the other two.
> 
> Upon reboot, it does this (from /boot) :
> 
> booting hd0a:/bsd: 8484712+2429968+244048+0+667648 [636809heap full 
> (0x9d304+65536)
> 
> And then reboots the system, as if from power-down/power-up.
> (already something I haven't seen before)
> 
> Reboot from "bsd.rd" and "bsd.sp", same results.  reboot from "obsd"
> (Sept 9), same results.  Not a kernel problem, it seems.  About this
> point, I'm starting to think how the serial console has let me down.
> 
> I remember how to bring up a DRAC remote CD image via ssh tunnels
> to the drac and how to run java in a windows browser, and
> reboot off the remote CD image, do another upgrade, all goes fine
> (again), but upon reboot, same results...  "heap full" and reboot.
> 
> Boot from remote CD, at the boot> prompt, enter "boot hd0a:/bsd",
> and it boots Just Fine from the local hard disk (only boot pulled
> from the remote CD).  Boot loader!  Reinstalled boot:
> 
> # installboot -v sd0
> Using / as root
> installing bootstrap on /dev/rsd0c
> using first-stage /usr/mdec/biosboot, second-stage /usr/mdec/boot
> copying /usr/mdec/boot to /boot
> /boot is 3 blocks x 32768 bytes
> fs block shift 3; part offset 64; inode block 24, offset 2088
> master boot record (MBR) at sector 0
> partition 3: type 0xA6 offset 64 size 2000397671
> /usr/mdec/biosboot will be written at sector 64
> 
> good, right?
> 
> Reboot off local hard disk, boom.  problem is still there.  maybe
> not the boot loader. :-/
> 
> Verified /boot on trouble system and good system are the same.  
> 
> I'm not going to cry "bug", since there are two nearly identical
> systems working just fine.  But I can't think of what I did wrong
> or what to do to fix it.
> 
> Suggestions?

You are hitting -DHEAP_LIMIT=0xA in /boot. The code is in libsa/alloa.c

No idea why. But something in that system is different. 

You do have one weird line in your disklabel output: a filesystem
mounted on swap? 

Can you boot into single user mode?

-Otto

> 
> Nick.
> 
> 
> $ dmesg
> OpenBSD 6.2-current (GENERIC.MP) #203: Sat Nov 11 19:01:19 MST 2017
> dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> real mem = 17131339776 (16337MB)
> avail mem = 16605306880 (15836MB)
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xe6680 (57 entries)
> bios0: vendor Dell Inc. version "2.8.0" date 06/24/2014
> bios0: Dell Inc. PowerEdge R210 II
> acpi0 at bios0: rev 2
> acpi0: sleep states S0 S4 S5
> acpi0: tables DSDT FACP SPMI ASF! HPET APIC MCFG BOOT SSDT ASPT SSDT SSDT 
> SPCR HEST ERST BERT EINJ
> acpi0: wakeup devices P0P1(S4) GLAN(S0) EHC1(S4) EHC2(S4) XHC_(S4) PXSX(S4) 
> RP01(S5) PXSX(S4) RP02(S5) PXSX(S4) RP03(S5) PXSX(S4) RP04(S5) PXSX(S4) 
> RP05(S5) PXSX(S4) [...]
> acpitimer0 at acpi0: 3579545 Hz, 24 bits
> acpihpet0 at acpi0: 14318179 Hz
> acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: Intel(R) Xeon(R) CPU E31230 @ 3.20GHz, 3193.24 MHz
> cpu0: 
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,ARAT
> cpu0: 256KB 64b/line 8-way L2 cache
> acpihpet0: recalibrated TSC frequency 3192748207 Hz
> cpu0: smt 0, core 0, package 0
> mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
> cpu0: apic clock running at 99MHz
> cpu0: mwait min=64, max=64, C-substates=0.2.1.1, IBE
> cpu1 at mainbus0: apid 1 (application processor)
> cpu1: Intel(R) Xeon(R) CPU E31230 @ 3.20GHz, 3192.75 MHz
> cpu1: 
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,ARAT
> cpu1: 256KB 64b/line 8-way L2 cache
> cpu1: smt 1, core 0, package 0
> cpu2 at mainbus0: apid 2 (application processor)
> cpu2: Intel(R) Xeon(R) CPU E31230 @ 3.20GHz, 3192.75 MHz
> cpu2: 
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,ARAT
> cpu2: 256KB 64b/line 8-way L2 cache
> cpu2: smt 0, core 1, package 0
> cpu3 at mainbus0: apid 3 (application processor)
> cpu3: Intel(R) Xeon(R) 

reboot loop on -current, one machine of several

[Nick Holland]

Help.

I was upgrading a few very similar machines to -current today.
ONE of the three decided to be unpleasant.  The thing has a
serial console, and but it is about 370km from me. :-/

Upgrade from Sep 9 current to today's current via bsd.rd, just
like the other two.

Upon reboot, it does this (from /boot) :

booting hd0a:/bsd: 8484712+2429968+244048+0+667648 [636809heap full 
(0x9d304+65536)

And then reboots the system, as if from power-down/power-up.
(already something I haven't seen before)

Reboot from "bsd.rd" and "bsd.sp", same results.  reboot from "obsd"
(Sept 9), same results.  Not a kernel problem, it seems.  About this
point, I'm starting to think how the serial console has let me down.

I remember how to bring up a DRAC remote CD image via ssh tunnels
to the drac and how to run java in a windows browser, and
reboot off the remote CD image, do another upgrade, all goes fine
(again), but upon reboot, same results...  "heap full" and reboot.

Boot from remote CD, at the boot> prompt, enter "boot hd0a:/bsd",
and it boots Just Fine from the local hard disk (only boot pulled
from the remote CD).  Boot loader!  Reinstalled boot:

# installboot -v sd0
Using / as root
installing bootstrap on /dev/rsd0c
using first-stage /usr/mdec/biosboot, second-stage /usr/mdec/boot
copying /usr/mdec/boot to /boot
/boot is 3 blocks x 32768 bytes
fs block shift 3; part offset 64; inode block 24, offset 2088
master boot record (MBR) at sector 0
partition 3: type 0xA6 offset 64 size 2000397671
/usr/mdec/biosboot will be written at sector 64

good, right?

Reboot off local hard disk, boom.  problem is still there.  maybe
not the boot loader. :-/

Verified /boot on trouble system and good system are the same.  

I'm not going to cry "bug", since there are two nearly identical
systems working just fine.  But I can't think of what I did wrong
or what to do to fix it.

Suggestions?

Nick.


$ dmesg
OpenBSD 6.2-current (GENERIC.MP) #203: Sat Nov 11 19:01:19 MST 2017
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 17131339776 (16337MB)
avail mem = 16605306880 (15836MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xe6680 (57 entries)
bios0: vendor Dell Inc. version "2.8.0" date 06/24/2014
bios0: Dell Inc. PowerEdge R210 II
acpi0 at bios0: rev 2
acpi0: sleep states S0 S4 S5
acpi0: tables DSDT FACP SPMI ASF! HPET APIC MCFG BOOT SSDT ASPT SSDT SSDT SPCR 
HEST ERST BERT EINJ
acpi0: wakeup devices P0P1(S4) GLAN(S0) EHC1(S4) EHC2(S4) XHC_(S4) PXSX(S4) 
RP01(S5) PXSX(S4) RP02(S5) PXSX(S4) RP03(S5) PXSX(S4) RP04(S5) PXSX(S4) 
RP05(S5) PXSX(S4) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpihpet0 at acpi0: 14318179 Hz
acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E31230 @ 3.20GHz, 3193.24 MHz
cpu0: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,ARAT
cpu0: 256KB 64b/line 8-way L2 cache
acpihpet0: recalibrated TSC frequency 3192748207 Hz
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 99MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.1, IBE
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Intel(R) Xeon(R) CPU E31230 @ 3.20GHz, 3192.75 MHz
cpu1: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,ARAT
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 1, core 0, package 0
cpu2 at mainbus0: apid 2 (application processor)
cpu2: Intel(R) Xeon(R) CPU E31230 @ 3.20GHz, 3192.75 MHz
cpu2: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,ARAT
cpu2: 256KB 64b/line 8-way L2 cache
cpu2: smt 0, core 1, package 0
cpu3 at mainbus0: apid 3 (application processor)
cpu3: Intel(R) Xeon(R) CPU E31230 @ 3.20GHz, 3192.75 MHz
cpu3: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,ARAT
cpu3: 256KB 64b/line 8-way L2 cache
cpu3: smt 1, core 1, package 0
cpu4 at mainbus0: apid 4 (application processor)
cpu4: Intel(R) Xeon(R) CPU E31230 @ 3.20GHz, 3192.75 MHz
cpu4: