Re: [EXTERNAL] Re: Troubleshooting pf congestion
> On 2020-09-14, Scott Reese wrote: >> Greetings: >> >> - Original Message - >>> From: "Uwe Werler" >>> To: "misc" , "Scott Reese" , "misc" >>> >>> Sent: Monday, September 14, 2020 12:47:31 PM >>> Subject: [EXTERNAL] Re: Troubleshooting pf congestion >> >>> Without seeing a rule set what should one say? >>> >> >>>> >>>>If anyone could spare a couple of sentences or a share a link to a page >>>>detailing what >>>>state causes the system to consider itself contested, I would >>>>appreciate it. >> >> Thanks for your reply. The question that I can't find a good answer for is, >> "What is pf congestion?". I would like to solve the actual problem myself, >> I'm >> just looking >> for some information about what it means for pf to be congested. > > When enqueueing packets to an interface fails (queue is full), a > global congestion marker variable in the kernel is set to the current > timestamp. > > When PF tests an inbound packet against rules (i.e. when it has a packet > that doesn't match an existing state) it checks if that congestion timestamp > is recent. If it is, the packet is dropped and the PF stats congestion > counter is incremented. > > Look around if_congested/if_congestion in /sys/net and the mq_ functions > in /sys/kern/uipc_mbuf.c - the functions described in mq_init(9) as "If the > queue is full then XX will be dropped" trigger congestion. > > You might get some suggestions if you post a description of your > configuration (at least which interface types - physical or virtual - > are in use, what they're connected to, what if any VPNs it's running, > and it may help to see the ruleset). > > Output from these might help too: > > netstat -m > systat mbuf | cat > vmstat -i > vmstat -m Greetings Stuart: Thank you for your reply. It was very helpful and pointed me in the right direction. The 1000+ windows workstations behind that firewall had been converted from 7 to 10. Most aren't allowed to access the internet, and the new OS is much more aggressive about trying to phone home. All of those dropped packets had to traverse all of the rules before being dropped, and that was the root cause of the issue. It didn't look like too much traffic because it was just SYN packets, but it was a lot of SYN packets. Again, thanks for your help. -Scott
Re: [EXTERNAL] Re: Troubleshooting pf congestion
On 2020-09-14, Scott Reese wrote: > Greetings: > > - Original Message - >> From: "Uwe Werler" >> To: "misc" , "Scott Reese" , "misc" >> >> Sent: Monday, September 14, 2020 12:47:31 PM >> Subject: [EXTERNAL] Re: Troubleshooting pf congestion > >> Without seeing a rule set what should one say? >> > >>> >>>If anyone could spare a couple of sentences or a share a link to a page >>>detailing what >>>state causes the system to consider itself contested, I would >>>appreciate it. > > Thanks for your reply. The question that I can't find a good answer for is, > "What is pf congestion?". I would like to solve the actual problem myself, > I'm just looking > for some information about what it means for pf to be congested. When enqueueing packets to an interface fails (queue is full), a global congestion marker variable in the kernel is set to the current timestamp. When PF tests an inbound packet against rules (i.e. when it has a packet that doesn't match an existing state) it checks if that congestion timestamp is recent. If it is, the packet is dropped and the PF stats congestion counter is incremented. Look around if_congested/if_congestion in /sys/net and the mq_ functions in /sys/kern/uipc_mbuf.c - the functions described in mq_init(9) as "If the queue is full then XX will be dropped" trigger congestion. You might get some suggestions if you post a description of your configuration (at least which interface types - physical or virtual - are in use, what they're connected to, what if any VPNs it's running, and it may help to see the ruleset). Output from these might help too: netstat -m systat mbuf | cat vmstat -i vmstat -m
Re: Troubleshooting pf congestion
Without seeing a rule set what should one say? Am 14. September 2020 15:19:46 GMT+00:00 schrieb Scott Reese : >Greetings: > >I am troubleshooting an issue: users complaining about network >performance. The firewall >is an OpenBSD 6.7 system with patches applied. I've traced the issue >and I'm seeing the >congestion counter incrementing on system. The problems that we're >seeing fit with what >I have been able to find about congestion - when the firewall is >congested it continues >passing packets that match existing state entries but it will not >create any new state >entries until the congestion clears. > >I'm having trouble troubleshooting it beyond that point because I have >not been able to >find any additional information about what the congestion counter is >counting. There is >the information in the pfctl man page: "congestion: network interface >queue congested", >but beyond that I can't really find any information about exactly what >network interface >queue is congested. > >I'm not seeing packets being dropped, either on the switch side or >firewall side that >correspond with the congestion counter going up. The average on the >congestion counter >stays around 10/s, but what it's really doing is going up by 100-300/s >for short periods >and then not moving for longer periods. > >If anyone could spare a couple of sentences or a share a link to a page >detailing what >state causes the system to consider itself contested, I would >appreciate it. > >Thanks for your time. > >-Scott > > >System dmesg: > >OpenBSD 6.7 (GENERIC.MP) #6: Thu Sep 3 14:08:18 MDT 2020 >r...@syspatch-67-amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP >real mem = 8386699264 (7998MB) >avail mem = 8119902208 (7743MB) >mpath0 at root >scsibus0 at mpath0: 256 targets >mainbus0 at root >bios0 at mainbus0: SMBIOS rev. 3.0 @ 0x7fb76000 (62 entries) >bios0: vendor American Megatrends Inc. version "2.2" date 05/23/2018 >bios0: Supermicro X11SSL-F >acpi0 at bios0: ACPI 5.0 >acpi0: sleep states S0 S4 S5 >acpi0: tables DSDT FACP APIC FPDT FIDT SPMI MCFG HPET LPIT SSDT SSDT >SSDT DBGP DBG2 SSDT SSDT UEFI SSDT DMAR EINJ ERST BERT HEST >acpi0: wakeup devices PEG0(S4) PEGP(S4) PEG1(S4) PEGP(S4) PEG2(S4) >PEGP(S4) RP09(S4) PXSX(S4) RP10(S4) PXSX(S4) RP11(S4) PXSX(S4) RP12(S4) >PXSX(S4) RP13(S4) PXSX(S4) [...] >acpitimer0 at acpi0: 3579545 Hz, 24 bits >acpimadt0 at acpi0 addr 0xfee0: PC-AT compat >cpu0 at mainbus0: apid 0 (boot processor) >cpu0: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3901.62 MHz, 06-9e-09 >cpu0: >FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN >cpu0: 256KB 64b/line 8-way L2 cache >cpu0: smt 0, core 0, package 0 >mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges >cpu0: apic clock running at 24MHz >cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4.1, IBE >cpu1 at mainbus0: apid 2 (application processor) >cpu1: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3900.01 MHz, 06-9e-09 >cpu1: >FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN >cpu1: 256KB 64b/line 8-way L2 cache >cpu1: smt 0, core 1, package 0 >cpu2 at mainbus0: apid 4 (application processor) >cpu2: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3900.01 MHz, 06-9e-09 >cpu2: >FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN >cpu2: 256KB 64b/line 8-way L2 cache >cpu2: smt 0, core 2, package 0 >cpu3 at mainbus0: apid 6 (application processor) >cpu3: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3900.01 MHz, 06-9e-09 >cpu3:
Re: Troubleshooting pf congestion
On Mon, Sep 14, 2020 at 11:19:46AM -0400, Scott Reese wrote: > Greetings: > > I am troubleshooting an issue: users complaining about network performance. > The firewall > is an OpenBSD 6.7 system with patches applied. I've traced the issue and I'm > seeing the > congestion counter incrementing on system. The problems that we're seeing fit > with what > I have been able to find about congestion - when the firewall is congested it > continues > passing packets that match existing state entries but it will not create any > new state > entries until the congestion clears. > > I'm having trouble troubleshooting it beyond that point because I have not > been able to > find any additional information about what the congestion counter is > counting. There is > the information in the pfctl man page: "congestion: network interface queue > congested", > but beyond that I can't really find any information about exactly what > network interface > queue is congested. > > I'm not seeing packets being dropped, either on the switch side or firewall > side that > correspond with the congestion counter going up. The average on the > congestion counter > stays around 10/s, but what it's really doing is going up by 100-300/s for > short periods > and then not moving for longer periods. > > If anyone could spare a couple of sentences or a share a link to a page > detailing what > state causes the system to consider itself contested, I would appreciate it. > > Thanks for your time. > > -Scott openbsd-archive.7691.n7.nabble.com/PF-congestion-question-td156490.html Description and potential remedy are stil true, afaik. -Otto > > > System dmesg: > > OpenBSD 6.7 (GENERIC.MP) #6: Thu Sep 3 14:08:18 MDT 2020 > > r...@syspatch-67-amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP > real mem = 8386699264 (7998MB) > avail mem = 8119902208 (7743MB) > mpath0 at root > scsibus0 at mpath0: 256 targets > mainbus0 at root > bios0 at mainbus0: SMBIOS rev. 3.0 @ 0x7fb76000 (62 entries) > bios0: vendor American Megatrends Inc. version "2.2" date 05/23/2018 > bios0: Supermicro X11SSL-F > acpi0 at bios0: ACPI 5.0 > acpi0: sleep states S0 S4 S5 > acpi0: tables DSDT FACP APIC FPDT FIDT SPMI MCFG HPET LPIT SSDT SSDT SSDT > DBGP DBG2 SSDT SSDT UEFI SSDT DMAR EINJ ERST BERT HEST > acpi0: wakeup devices PEG0(S4) PEGP(S4) PEG1(S4) PEGP(S4) PEG2(S4) PEGP(S4) > RP09(S4) PXSX(S4) RP10(S4) PXSX(S4) RP11(S4) PXSX(S4) RP12(S4) PXSX(S4) > RP13(S4) PXSX(S4) [...] > acpitimer0 at acpi0: 3579545 Hz, 24 bits > acpimadt0 at acpi0 addr 0xfee0: PC-AT compat > cpu0 at mainbus0: apid 0 (boot processor) > cpu0: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3901.62 MHz, 06-9e-09 > cpu0: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN > cpu0: 256KB 64b/line 8-way L2 cache > cpu0: smt 0, core 0, package 0 > mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges > cpu0: apic clock running at 24MHz > cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4.1, IBE > cpu1 at mainbus0: apid 2 (application processor) > cpu1: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3900.01 MHz, 06-9e-09 > cpu1: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN > cpu1: 256KB 64b/line 8-way L2 cache > cpu1: smt 0, core 1, package 0 > cpu2 at mainbus0: apid 4 (application processor) > cpu2: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3900.01 MHz, 06-9e-09 > cpu2: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN > cpu2: 256KB 64b/line 8-way L2 cache > cpu2: smt 0, core 2, package 0 > cpu3 at mainbus0: apid 6 (application processor) > cpu3: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3900.01 MHz, 06-9e-09 > cpu3:
Re: [EXTERNAL] Re: Troubleshooting pf congestion
Greetings: - Original Message - > From: "Uwe Werler" > To: "misc" , "Scott Reese" , "misc" > > Sent: Monday, September 14, 2020 12:47:31 PM > Subject: [EXTERNAL] Re: Troubleshooting pf congestion > Without seeing a rule set what should one say? > >> >>If anyone could spare a couple of sentences or a share a link to a page >>detailing what >>state causes the system to consider itself contested, I would >>appreciate it. Thanks for your reply. The question that I can't find a good answer for is, "What is pf congestion?". I would like to solve the actual problem myself, I'm just looking for some information about what it means for pf to be congested. -Scott
Troubleshooting pf congestion
Greetings: I am troubleshooting an issue: users complaining about network performance. The firewall is an OpenBSD 6.7 system with patches applied. I've traced the issue and I'm seeing the congestion counter incrementing on system. The problems that we're seeing fit with what I have been able to find about congestion - when the firewall is congested it continues passing packets that match existing state entries but it will not create any new state entries until the congestion clears. I'm having trouble troubleshooting it beyond that point because I have not been able to find any additional information about what the congestion counter is counting. There is the information in the pfctl man page: "congestion: network interface queue congested", but beyond that I can't really find any information about exactly what network interface queue is congested. I'm not seeing packets being dropped, either on the switch side or firewall side that correspond with the congestion counter going up. The average on the congestion counter stays around 10/s, but what it's really doing is going up by 100-300/s for short periods and then not moving for longer periods. If anyone could spare a couple of sentences or a share a link to a page detailing what state causes the system to consider itself contested, I would appreciate it. Thanks for your time. -Scott System dmesg: OpenBSD 6.7 (GENERIC.MP) #6: Thu Sep 3 14:08:18 MDT 2020 r...@syspatch-67-amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 8386699264 (7998MB) avail mem = 8119902208 (7743MB) mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 3.0 @ 0x7fb76000 (62 entries) bios0: vendor American Megatrends Inc. version "2.2" date 05/23/2018 bios0: Supermicro X11SSL-F acpi0 at bios0: ACPI 5.0 acpi0: sleep states S0 S4 S5 acpi0: tables DSDT FACP APIC FPDT FIDT SPMI MCFG HPET LPIT SSDT SSDT SSDT DBGP DBG2 SSDT SSDT UEFI SSDT DMAR EINJ ERST BERT HEST acpi0: wakeup devices PEG0(S4) PEGP(S4) PEG1(S4) PEGP(S4) PEG2(S4) PEGP(S4) RP09(S4) PXSX(S4) RP10(S4) PXSX(S4) RP11(S4) PXSX(S4) RP12(S4) PXSX(S4) RP13(S4) PXSX(S4) [...] acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3901.62 MHz, 06-9e-09 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN cpu0: 256KB 64b/line 8-way L2 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges cpu0: apic clock running at 24MHz cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4.1, IBE cpu1 at mainbus0: apid 2 (application processor) cpu1: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3900.01 MHz, 06-9e-09 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN cpu1: 256KB 64b/line 8-way L2 cache cpu1: smt 0, core 1, package 0 cpu2 at mainbus0: apid 4 (application processor) cpu2: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3900.01 MHz, 06-9e-09 cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN cpu2: 256KB 64b/line 8-way L2 cache cpu2: smt 0, core 2, package 0 cpu3 at mainbus0: apid 6 (application processor) cpu3: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3900.01 MHz, 06-9e-09 cpu3:
