Re: Dell R630 high interrupts on acpi0
On 17.12.2014. 6:34, Philip Guenther wrote: Uh, ACPI *requires* that C1 exist. The halt instruction is defined as entering C1, so not having C1 would mean your CPU lacks a basic manadatory ia32 instruction. Hopefully the BIOS docs explain that you're just disabling "deep" C-states or something like that. If not, yell at the company that made it. With the exception of "C1E", I wouldn't tell a BIOS to disable C-states unless it was causing the OS to have a problem or you're actively trying to use the computer to heat your house. "C1E" is a cross between C1 and C3; the issue is that bugs in multiple early hardware implementations mean it'll behave poorly depending on exactly how the OS handles it. This is something to test...and then test again with each release you install... Thank you, this is very informative. I have disabled C states because I have seen ifq.drops and netlivelocks when I run udp tcpbench over ix interface and C-states are enabled. This is between two Dell R620. With Dell R630 I have bigger problems so i can't test them right now. tcpbench client sends around 350kpps for 20s tcpbench server gets around 212kpps tcpbench udp server drops: Without C states and C1E: net.inet.ip.ifq.maxlen=2048 net.inet.ip.ifq.drops=0 kern.netlivelocks=3 With C states and C1E (DAPC): net.inet.ip.ifq.maxlen=2048 net.inet.ip.ifq.drops=19143 kern.netlivelocks=86 so I explained to myself that C states are bad for packet bursts and these boxes will be uber high speed firewalls :) and of course after reading you mail I only disable C1E and everything is fine. With C states and without C1E net.inet.ip.ifq.maxlen=2048 net.inet.ip.ifq.drops=0 kern.netlivelocks=3 Monitor/Mwait - Disabled I would suggest leaving that on. We ain't using it *right now*, but, well, the source tree on my laptop is, and more than ever. :-) mwait is enabled :)
Re: Dell R630 high interrupts on acpi0
> Date: Fri, 19 Dec 2014 23:41:56 +1000
> From: Jonathan Matthew
>
> On Thu, Dec 18, 2014 at 11:14:54PM +0100, Frederic Nowak wrote:
> > Hi!
> >
> > The diff for extracting memory ranges from ACPI was obviously not tested
> > with ACPI disabled...so we definitely have to check if the values in
> > pcimem_range make some sense. The diff below now uses the old values in
> > case ACPI is disabled or does not return valid values.
> >
> > I hope this is somewhat interesting for someone else as well :)
> > Please let me know if this is just noise or if there is anything else I
> > am missing.
>
> This looks like the right direction, except I think we only really want to use
> the ACPI range information to add new ranges to the existing one that covers
> the 36-bit address space, rather than relying on it for everything. No point
> relying on the ACPI stuff being correct if we don't need to.
>
> I hacked this around a bit today and ended up with the diff below. This only
> deals with ranges outside the 36-bit space, so it makes no difference on most
> machines, and on the r630 that needs it, it adds two ranges,
> 0x0300 to 0x033F and 0x0340 to
> 0x037F.
This is going in the right direction. The way I designed things
though was that the acpi code would build up the complete extent
purely from information provided by _CRS, and set pcimem_ex before
pci_init_extents() gets called.
> Index: arch/amd64/pci/pci_machdep.c
> ===
> RCS file: /cvs/src/sys/arch/amd64/pci/pci_machdep.c,v
> retrieving revision 1.60
> diff -u -p -r1.60 pci_machdep.c
> --- arch/amd64/pci/pci_machdep.c 16 Dec 2014 23:13:20 - 1.60
> +++ arch/amd64/pci/pci_machdep.c 19 Dec 2014 12:54:44 -
> @@ -90,12 +90,17 @@
> #include
>
> #include "ioapic.h"
> +#include "acpi.h"
>
> #if NIOAPIC > 0
> #include
> #include
> #endif
>
> +#if NACPI > 0
> +#include
> +#endif
> +
> /*
> * Memory Mapped Configuration space access.
> *
> @@ -622,17 +627,13 @@ pci_init_extents(void)
>* here. As long as vendors continue to support
>* 32-bit operating systems, we should never see BARs
>* outside that region.
> - *
> - * Dell 13G servers have important devices outside the
> - * 36-bit address space. Until we can extract the address
> - * ranges from ACPI, expand the allowed range to suit.
>*/
> pcimem_ex = extent_create("pcimem", 0, 0xUL,
> M_DEVBUF, NULL, 0, EX_NOWAIT);
> if (pcimem_ex == NULL)
> return;
> - extent_alloc_region(pcimem_ex, 0x400UL,
> - 0xfc00UL, EX_NOWAIT);
> + extent_alloc_region(pcimem_ex, 0x10UL,
> + 0xfff0UL, EX_NOWAIT);
>
> for (bmp = bios_memmap; bmp->type != BIOS_MAP_END; bmp++) {
> /*
> @@ -657,6 +658,14 @@ pci_init_extents(void)
> /* Take out the video buffer area and BIOS areas. */
> extent_alloc_region(pcimem_ex, IOM_BEGIN, IOM_SIZE,
> EX_CONFLICTOK | EX_NOWAIT);
> +
> +#if NACPI > 0
> + /*
> + * Free up any regions outside the 36-bit address space
> + * specified via ACPI.
> + */
> + acpi_pciroots_extents(pcimem_ex, 0x10UL);
> +#endif
> }
>
> if (pcibus_ex == NULL) {
> @@ -665,7 +674,6 @@ pci_init_extents(void)
> }
> }
>
> -#include "acpi.h"
> #if NACPI > 0
> void acpi_pci_match(struct device *, struct pci_attach_args *);
> pcireg_t acpi_pci_min_powerstate(pci_chipset_tag_t, pcitag_t);
> Index: dev/acpi/acpivar.h
> ===
> RCS file: /cvs/src/sys/dev/acpi/acpivar.h,v
> retrieving revision 1.79
> diff -u -p -r1.79 acpivar.h
> --- dev/acpi/acpivar.h8 Dec 2014 07:12:37 - 1.79
> +++ dev/acpi/acpivar.h19 Dec 2014 12:54:44 -
> @@ -47,6 +47,7 @@ extern u_int8_t acpi_lapic_flags[LAPIC_M
> struct klist;
> struct acpiec_softc;
> struct acpipwrres_softc;
> +struct extent;
>
> struct acpivideo_softc {
> struct device sc_dev;
> @@ -357,6 +358,7 @@ int acpi_acquire_glk(uint32_t *);
> int acpi_release_glk(uint32_t *);
>
> void acpi_pciroots_attach(struct device *, void *, cfprint_t);
> +void acpi_pciroots_extents(struct extent *, u_int64_t);
>
> #endif
>
> Index: dev/acpi/amltypes.h
> ===
> RCS file: /cvs/src/sys/dev/acpi/amltypes.h,v
> retrieving revision 1.40
> diff -u -p -r1.40 amltypes.h
> --- dev/acpi/amltypes.h 7 Sep 2012 19:19:59 - 1.40
> +++ dev/acpi/amltypes.h 19 Dec 2014 12:54:44 -
> @@ -346,6 +346,12 @@ struct a
Re: Dell R630 high interrupts on acpi0
On Thu, Dec 18, 2014 at 11:14:54PM +0100, Frederic Nowak wrote:
> Hi!
>
> The diff for extracting memory ranges from ACPI was obviously not tested
> with ACPI disabled...so we definitely have to check if the values in
> pcimem_range make some sense. The diff below now uses the old values in
> case ACPI is disabled or does not return valid values.
>
> I hope this is somewhat interesting for someone else as well :)
> Please let me know if this is just noise or if there is anything else I
> am missing.
This looks like the right direction, except I think we only really want to use
the ACPI range information to add new ranges to the existing one that covers
the 36-bit address space, rather than relying on it for everything. No point
relying on the ACPI stuff being correct if we don't need to.
I hacked this around a bit today and ended up with the diff below. This only
deals with ranges outside the 36-bit space, so it makes no difference on most
machines, and on the r630 that needs it, it adds two ranges,
0x0300 to 0x033F and 0x0340 to
0x037F.
Index: arch/amd64/pci/pci_machdep.c
===
RCS file: /cvs/src/sys/arch/amd64/pci/pci_machdep.c,v
retrieving revision 1.60
diff -u -p -r1.60 pci_machdep.c
--- arch/amd64/pci/pci_machdep.c16 Dec 2014 23:13:20 - 1.60
+++ arch/amd64/pci/pci_machdep.c19 Dec 2014 12:54:44 -
@@ -90,12 +90,17 @@
#include
#include "ioapic.h"
+#include "acpi.h"
#if NIOAPIC > 0
#include
#include
#endif
+#if NACPI > 0
+#include
+#endif
+
/*
* Memory Mapped Configuration space access.
*
@@ -622,17 +627,13 @@ pci_init_extents(void)
* here. As long as vendors continue to support
* 32-bit operating systems, we should never see BARs
* outside that region.
-*
-* Dell 13G servers have important devices outside the
-* 36-bit address space. Until we can extract the address
-* ranges from ACPI, expand the allowed range to suit.
*/
pcimem_ex = extent_create("pcimem", 0, 0xUL,
M_DEVBUF, NULL, 0, EX_NOWAIT);
if (pcimem_ex == NULL)
return;
- extent_alloc_region(pcimem_ex, 0x400UL,
- 0xfc00UL, EX_NOWAIT);
+ extent_alloc_region(pcimem_ex, 0x10UL,
+ 0xfff0UL, EX_NOWAIT);
for (bmp = bios_memmap; bmp->type != BIOS_MAP_END; bmp++) {
/*
@@ -657,6 +658,14 @@ pci_init_extents(void)
/* Take out the video buffer area and BIOS areas. */
extent_alloc_region(pcimem_ex, IOM_BEGIN, IOM_SIZE,
EX_CONFLICTOK | EX_NOWAIT);
+
+#if NACPI > 0
+ /*
+* Free up any regions outside the 36-bit address space
+* specified via ACPI.
+*/
+ acpi_pciroots_extents(pcimem_ex, 0x10UL);
+#endif
}
if (pcibus_ex == NULL) {
@@ -665,7 +674,6 @@ pci_init_extents(void)
}
}
-#include "acpi.h"
#if NACPI > 0
void acpi_pci_match(struct device *, struct pci_attach_args *);
pcireg_t acpi_pci_min_powerstate(pci_chipset_tag_t, pcitag_t);
Index: dev/acpi/acpivar.h
===
RCS file: /cvs/src/sys/dev/acpi/acpivar.h,v
retrieving revision 1.79
diff -u -p -r1.79 acpivar.h
--- dev/acpi/acpivar.h 8 Dec 2014 07:12:37 - 1.79
+++ dev/acpi/acpivar.h 19 Dec 2014 12:54:44 -
@@ -47,6 +47,7 @@ extern u_int8_t acpi_lapic_flags[LAPIC_M
struct klist;
struct acpiec_softc;
struct acpipwrres_softc;
+struct extent;
struct acpivideo_softc {
struct device sc_dev;
@@ -357,6 +358,7 @@ int acpi_acquire_glk(uint32_t *);
intacpi_release_glk(uint32_t *);
void acpi_pciroots_attach(struct device *, void *, cfprint_t);
+void acpi_pciroots_extents(struct extent *, u_int64_t);
#endif
Index: dev/acpi/amltypes.h
===
RCS file: /cvs/src/sys/dev/acpi/amltypes.h,v
retrieving revision 1.40
diff -u -p -r1.40 amltypes.h
--- dev/acpi/amltypes.h 7 Sep 2012 19:19:59 - 1.40
+++ dev/acpi/amltypes.h 19 Dec 2014 12:54:44 -
@@ -346,6 +346,12 @@ struct aml_value {
#define aml_pkglen(v) ((v)->length)
#define aml_pkgval(v,i)(&(v)->v_package[(i)])
+struct acpi_pci_range {
+ SLIST_ENTRY(acpi_pci_range) next;
+ uint64_tbase;
+ uint64_tlen;
+};
+
struct acpi_pci {
TAILQ_ENTRY(acpi_pci) next;
@@ -362,6 +368,8 @@ struct acpi_pci {
int _s3w;
int _s4d;
int
Re: Dell R630 high interrupts on acpi0
Hi!
The diff for extracting memory ranges from ACPI was obviously not tested
with ACPI disabled...so we definitely have to check if the values in
pcimem_range make some sense. The diff below now uses the old values in
case ACPI is disabled or does not return valid values.
I hope this is somewhat interesting for someone else as well :)
Please let me know if this is just noise or if there is anything else I
am missing.
Index: sys/arch/amd64/include/pci_machdep.h
===
RCS file: /cvs/src/sys/arch/amd64/include/pci_machdep.h,v
retrieving revision 1.22
diff -u -p -b -r1.22 pci_machdep.h
--- sys/arch/amd64/include/pci_machdep.h6 Nov 2013 10:40:36 -
1.22
+++ sys/arch/amd64/include/pci_machdep.h17 Dec 2014 06:00:01 -
@@ -64,6 +64,7 @@ extern int pci_mcfg_min_bus, pci_mcfg_ma
struct pci_attach_args;
+extern uint64_t pcimem_range[2];
extern struct extent *pciio_ex;
extern struct extent *pcimem_ex;
extern struct extent *pcibus_ex;
Index: sys/arch/amd64/pci/pci_machdep.c
===
RCS file: /cvs/src/sys/arch/amd64/pci/pci_machdep.c,v
retrieving revision 1.60
diff -u -p -b -r1.60 pci_machdep.c
--- sys/arch/amd64/pci/pci_machdep.c16 Dec 2014 23:13:20 - 1.60
+++ sys/arch/amd64/pci/pci_machdep.c18 Dec 2014 22:01:29 -
@@ -66,6 +66,7 @@
*/
#include
+#include
#include
#include
#include
@@ -592,6 +593,7 @@ pci_intr_disestablish(pci_chipset_tag_t
intr_disestablish(cookie);
}
+uint64_t pcimem_range[2] = {UINT64_MAX, 0};
struct extent *pciio_ex;
struct extent *pcimem_ex;
struct extent *pcibus_ex;
@@ -618,31 +620,32 @@ pci_init_extents(void)
if (pcimem_ex == NULL) {
/*
-* Cover the 36-bit address space addressable by PAE
-* here. As long as vendors continue to support
-* 32-bit operating systems, we should never see BARs
-* outside that region.
-*
-* Dell 13G servers have important devices outside the
-* 36-bit address space. Until we can extract the address
-* ranges from ACPI, expand the allowed range to suit.
+* Cover the address space extracted from
+* ACPI. Fallback to fixed ranges in case ACPI is
+* disabled or reporting an invalid address range.
*/
+ if(pcimem_range[0] >= pcimem_range[1]) {
+ pcimem_range[0] = 0;
+ pcimem_range[1] = 0x400UL;
+ }
+
pcimem_ex = extent_create("pcimem", 0, 0xUL,
M_DEVBUF, NULL, 0, EX_NOWAIT);
if (pcimem_ex == NULL)
return;
- extent_alloc_region(pcimem_ex, 0x400UL,
- 0xfc00UL, EX_NOWAIT);
+ extent_alloc_region(pcimem_ex, pcimem_range[1] + 1,
+ (0xUL - pcimem_range[1]), EX_NOWAIT);
for (bmp = bios_memmap; bmp->type != BIOS_MAP_END; bmp++) {
/*
-* Ignore address space beyond 4G.
+* Ignore address space beyond fixed/extracted
+* address range.
*/
- if (bmp->addr >= 0x1ULL)
+ if (bmp->addr > pcimem_range[1])
continue;
size = bmp->size;
- if (bmp->addr + size >= 0x1ULL)
- size = 0x1ULL - bmp->addr;
+ if (bmp->addr + size > pcimem_range[1])
+ size = pcimem_range[1] - bmp->addr + 1;
/* Ignore zero-sized regions. */
if (size == 0)
Index: sys/dev/acpi/acpi.c
===
RCS file: /cvs/src/sys/dev/acpi/acpi.c,v
retrieving revision 1.277
diff -u -p -b -r1.277 acpi.c
--- sys/dev/acpi/acpi.c 9 Dec 2014 06:58:29 - 1.277
+++ sys/dev/acpi/acpi.c 17 Dec 2014 05:59:36 -
@@ -385,0 +385,0 @@ TAILQ_HEAD(, acpi_pci) acpi_pcirootdevs
int acpi_getpci(struct aml_node *node, void *arg);
int acpi_getminbus(union acpi_resource *crs, void *arg);
+int acpi_getmemrange(union acpi_resource *crs, void *arg);
+
+int
+acpi_getmemrange(union acpi_resource *crs, void *arg)
+{
+ int typ = AML_CRSTYPE(crs);
+ uint64_t *range = arg; /* size 2 */
+ uint64_t min, max;
+
+ switch(typ) {
+ case LR_24BIT:
+ min = crs->lr_m24._min;
+ max = crs->lr_m24._max;
+ break;
+ case LR_32BIT:
+ case LR_32BITFIXED:
+ min = crs->lr_m32._min;
+ max = crs->lr_m32
Re: Dell R630 high interrupts on acpi0
> Date: Tue, 16 Dec 2014 21:34:29 -0800 > From: Philip Guenther > > > R620 have similar settings and can't see C states in dmesg > > acpicpu0 at acpi0 > > That's either insane, or a bug in our acpicpu code, IMO. Probably just the effect of the BIOS not advertising C states at all. Like you said, C1 support is mandatory, so there is no real need to advertise it unless some of the deeper states are advertised as well.
Re: Dell R630 high interrupts on acpi0
> On 16 dec 2014, at 06:40, David Gwynne wrote: > > others have hit this on r620s as well I donât see it on mine. interrupt total rate irq0/clock 9587998940 1599 irq0/ipi136166514 22 irq144/acpi020 irq112/ix029053603446 4847 irq113/ix127844456217 4646 irq96/mfi080725871 irq114/ubsec0 3101629892 517 irq98/ehci0 1120 irq115/em0 4928262870 822 irq116/em1 211437268 35 irq99/ehci1280 irq100/ahci010 Total 7487162787712493 This is a pre-5.6 OpenBSD 5.6-current (GENERIC.MP) #394: Wed Oct 1 12:54:54 MDT 2014 dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 8492285952 (8098MB) avail mem = 8257511424 (7874MB) mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xcf42c000 (99 entries) bios0: vendor Dell Inc. version "1.3.6" date 09/11/2012 bios0: Dell Inc. PowerEdge R620 acpi0 at bios0: rev 2 acpi0: sleep states S0 S4 S5 acpi0: tables DSDT FACP APIC SPCR HPET DMAR MCFG WD__ SLIC ERST HEST BERT EINJ TCPA PC__ SRAT SSDT acpi0: wakeup devices PCI0(S5) PCI1(S5) acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Xeon(R) CPU E5-2643 0 @ 3.30GHz, 3400.43 MHz cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX ,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLI NE,AES,XSAVE,AVX,NXE,PAGE1GB,LONG,LAHF,PERF,ITSC cpu0: 256KB 64b/line 8-way L2 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges cpu0: apic clock running at 99MHz cpu0: mwait min=64, max=64, C-substates=0.2.1.1.2, IBE cpu1 at mainbus0: apid 32 (application processor) cpu1: Intel(R) Xeon(R) CPU E5-2643 0 @ 3.30GHz, 3400.00 MHz cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX ,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLI NE,AES,XSAVE,AVX,NXE,PAGE1GB,LONG,LAHF,PERF,ITSC cpu1: 256KB 64b/line 8-way L2 cache cpu1: smt 0, core 0, package 1 cpu2 at mainbus0: apid 2 (application processor) cpu2: Intel(R) Xeon(R) CPU E5-2643 0 @ 3.30GHz, 3400.00 MHz cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX ,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLI NE,AES,XSAVE,AVX,NXE,PAGE1GB,LONG,LAHF,PERF,ITSC cpu2: 256KB 64b/line 8-way L2 cache cpu2: smt 0, core 1, package 0 cpu3 at mainbus0: apid 34 (application processor) cpu3: Intel(R) Xeon(R) CPU E5-2643 0 @ 3.30GHz, 3400.00 MHz cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX ,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLI NE,AES,XSAVE,AVX,NXE,PAGE1GB,LONG,LAHF,PERF,ITSC cpu3: 256KB 64b/line 8-way L2 cache cpu3: smt 0, core 1, package 1 cpu4 at mainbus0: apid 4 (application processor) cpu4: Intel(R) Xeon(R) CPU E5-2643 0 @ 3.30GHz, 3400.00 MHz cpu4: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX ,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLI NE,AES,XSAVE,AVX,NXE,PAGE1GB,LONG,LAHF,PERF,ITSC cpu4: 256KB 64b/line 8-way L2 cache cpu4: smt 0, core 2, package 0 cpu5 at mainbus0: apid 36 (application processor) cpu5: Intel(R) Xeon(R) CPU E5-2643 0 @ 3.30GHz, 3400.00 MHz cpu5: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX ,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLI NE,AES,XSAVE,AVX,NXE,PAGE1GB,LONG,LAHF,PERF,ITSC cpu5: 256KB 64b/line 8-way L2 cache cpu5: smt 0, core 2, package 1 cpu6 at mainbus0: apid 6 (application processor) cpu6: Intel(R) Xeon(R) CPU E5-2643 0 @ 3.30GHz, 3400.00 MHz cpu6: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX ,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLI NE,AES,XSAVE,AVX,NXE,PAGE1GB,LONG,LAHF,PERF,ITSC cpu6: 256KB 64b/line 8-way L2 cache cpu6: smt 0, core 3, package 0 cpu7 at mainbus0: apid 38 (application processor) cpu7: Intel(R) Xeon(R) CPU E5-2643 0 @ 3.30GHz, 340
Re: Dell R630 high interrupts on acpi0
Hi!
The below diff extracts the memory range information from ACPI. It looks
up all the memory ranges in _CRS and calculates minimal and maximal
values for pci_machdep.c.
I tested this on two amd64 machines and see no difference in pcidump.
Do you think we need to keep the old method in case the ACPI on some
machines does not report correct memory ranges? A simple check would be
to test if pcimem_range[0] < pcimem_range[1]...
Index: sys/arch/amd64/include/pci_machdep.h
===
RCS file: /cvs/src/sys/arch/amd64/include/pci_machdep.h,v
retrieving revision 1.22
diff -u -p -b -r1.22 pci_machdep.h
--- sys/arch/amd64/include/pci_machdep.h6 Nov 2013 10:40:36 -
1.22
+++ sys/arch/amd64/include/pci_machdep.h17 Dec 2014 06:00:01 -
@@ -64,6 +64,7 @@ extern int pci_mcfg_min_bus, pci_mcfg_ma
struct pci_attach_args;
+extern uint64_t pcimem_range[2];
extern struct extent *pciio_ex;
extern struct extent *pcimem_ex;
extern struct extent *pcibus_ex;
Index: sys/arch/amd64/pci/pci_machdep.c
===
RCS file: /cvs/src/sys/arch/amd64/pci/pci_machdep.c,v
retrieving revision 1.60
diff -u -p -b -r1.60 pci_machdep.c
--- sys/arch/amd64/pci/pci_machdep.c16 Dec 2014 23:13:20 - 1.60
+++ sys/arch/amd64/pci/pci_machdep.c17 Dec 2014 07:16:47 -
@@ -66,6 +66,7 @@
*/
#include
+#include
#include
#include
#include
@@ -592,6 +593,7 @@ pci_intr_disestablish(pci_chipset_tag_t
intr_disestablish(cookie);
}
+uint64_t pcimem_range[2] = {UINT64_MAX, 0};
struct extent *pciio_ex;
struct extent *pcimem_ex;
struct extent *pcibus_ex;
@@ -618,31 +620,25 @@ pci_init_extents(void)
if (pcimem_ex == NULL) {
/*
-* Cover the 36-bit address space addressable by PAE
-* here. As long as vendors continue to support
-* 32-bit operating systems, we should never see BARs
-* outside that region.
-*
-* Dell 13G servers have important devices outside the
-* 36-bit address space. Until we can extract the address
-* ranges from ACPI, expand the allowed range to suit.
+* Cover the address space extracted from ACPI.
*/
pcimem_ex = extent_create("pcimem", 0, 0xUL,
M_DEVBUF, NULL, 0, EX_NOWAIT);
if (pcimem_ex == NULL)
return;
- extent_alloc_region(pcimem_ex, 0x400UL,
- 0xfc00UL, EX_NOWAIT);
+ extent_alloc_region(pcimem_ex, pcimem_range[1] + 1,
+ (0xUL - pcimem_range[1]), EX_NOWAIT);
for (bmp = bios_memmap; bmp->type != BIOS_MAP_END; bmp++) {
/*
-* Ignore address space beyond 4G.
+* Ignore address space beyond address range
+* extracted from ACPI.
*/
- if (bmp->addr >= 0x1ULL)
+ if (bmp->addr > pcimem_range[1])
continue;
size = bmp->size;
- if (bmp->addr + size >= 0x1ULL)
- size = 0x1ULL - bmp->addr;
+ if (bmp->addr + size > pcimem_range[1])
+ size = pcimem_range[1] - bmp->addr + 1;
/* Ignore zero-sized regions. */
if (size == 0)
Index: sys/dev/acpi/acpi.c
===
RCS file: /cvs/src/sys/dev/acpi/acpi.c,v
retrieving revision 1.277
diff -u -p -b -r1.277 acpi.c
--- sys/dev/acpi/acpi.c 9 Dec 2014 06:58:29 - 1.277
+++ sys/dev/acpi/acpi.c 17 Dec 2014 05:59:36 -
@@ -385,0 +385,0 @@ TAILQ_HEAD(, acpi_pci) acpi_pcirootdevs
int acpi_getpci(struct aml_node *node, void *arg);
int acpi_getminbus(union acpi_resource *crs, void *arg);
+int acpi_getmemrange(union acpi_resource *crs, void *arg);
+
+int
+acpi_getmemrange(union acpi_resource *crs, void *arg)
+{
+ int typ = AML_CRSTYPE(crs);
+ uint64_t *range = arg; /* size 2 */
+ uint64_t min, max;
+
+ switch(typ) {
+ case LR_24BIT:
+ min = crs->lr_m24._min;
+ max = crs->lr_m24._max;
+ break;
+ case LR_32BIT:
+ case LR_32BITFIXED:
+ min = crs->lr_m32._min;
+ max = crs->lr_m32._max;
+ break;
+ case LR_WORD:
+ min = crs->lr_word._min;
+ max = crs->lr_word._max;
+ break;
+ case LR_DWORD:
+ min = crs->lr_dword._min;
+ max = crs->lr_dword._max;
+ break;
+
Re: Dell R630 high interrupts on acpi0
On Tue, Dec 16, 2014 at 2:45 PM, Hrvoje Popovski wrote: > on R630 i have custom bios settings and noticed that even if C states are > disabled in bios i can see them in dmesg > acpicpu0 at acpi0: C1 Uh, ACPI *requires* that C1 exist. The halt instruction is defined as entering C1, so not having C1 would mean your CPU lacks a basic manadatory ia32 instruction. Hopefully the BIOS docs explain that you're just disabling "deep" C-states or something like that. If not, yell at the company that made it. With the exception of "C1E", I wouldn't tell a BIOS to disable C-states unless it was causing the OS to have a problem or you're actively trying to use the computer to heat your house. "C1E" is a cross between C1 and C3; the issue is that bugs in multiple early hardware implementations mean it'll behave poorly depending on exactly how the OS handles it. This is something to test...and then test again with each release you install... > X2Apic is disabled too but in dmesg i see > cpu0: FPU,CPI,...SSE4.1,SSE4.2,x2APIC That just means the CPU has the feature bit set in the CPUID sets. The BIOS is presumably configuring ACPI (which is what OpenBSD pays attention to) to use the original LAPIC tables instead of the x2APIC tables for locating CPUs and interrupts. > R620 have similar settings and can't see C states in dmesg > acpicpu0 at acpi0 That's either insane, or a bug in our acpicpu code, IMO. > R620 bios settings ... > Monitor/Mwait - Disabled I would suggest leaving that on. We ain't using it *right now*, but, well, the source tree on my laptop is, and more than ever. :-) Philip Guenther
Re: Dell R630 high interrupts on acpi0
On 16.12.2014. 6:16, Jonathan Matthew wrote: We just got some r630s too, so I spent some time last week figuring out what's going on here. Something in the AML wants to talk to the intel MEI device. Normally this works, but on the new generation of dell machines (we've seen it on r630s and r730s), it's been moved outside the pci memory range we currently allow on amd64. You can see this in your dmesgs: 0:22:0: mem address conflict 0x3303000/0x10 0:22:1: mem address conflict 0x3302000/0x10 The interrupt will keep triggering until it manages to talk to the device, which will never happen. kettenis@ says we can get the pci memory range information we need to deal with this from acpi. Until that happens, expanding the allowed pci memory range makes things work properly. Hi, on R630 i have custom bios settings and noticed that even if C states are disabled in bios i can see them in dmesg acpicpu0 at acpi0: C1 acpicpu1 at acpi0: C1 acpicpu2 at acpi0: C1 acpicpu3 at acpi0: C1 acpicpu4 at acpi0: C1 acpicpu5 at acpi0: C1 acpicpu6 at acpi0: C1 acpicpu7 at acpi0: C1 X2Apic is disabled too but in dmesg i see cpu0: FPU,CPI,...SSE4.1,SSE4.2,x2APIC This is not good, right? R630 bios settings Processor Settings: Logical Processor - Disabled QPI Speed - Maximum data rate Alternate RTID Settings - Disabled Virtualization Technology - Disabled Address Translation Service (ATS) - Disabled (gray) Adjacent Cache Line Prefetch - Enabled Hardware Prefetcher - Enabled DCU Streamer Prefetcher - Enabled DCU IP Prefetcher - Enabled Execute Disabled - Enabled Logical Processor Idling - Disabled Configurable TDP - Nominal X2Apic Mode - Disabled (gray) Dell Controlled Turbo - Enabled System Profile Settings: System Profile - Custom CPU Power Management - Maximum Performance Memory Frequency - Maximum Performance Turbo Boost - Enabled Energy Efficient Turbo - Disabled C1E - Disabled C states - Disabled Collaborative CPU Performance Control - Disabled Memory Patrol Scrub - Standard Memory Refresh Rate - 1x Uncore Frequency - Maximum Energy Efficient Policy - Performance Full dmesg and acpidump from R630 http://kosjenka.srce.hr/~hrvoje/R630_custom_dmesg.txt http://kosjenka.srce.hr/~hrvoje/R630_custom.tgz R620 have similar settings and can't see C states in dmesg acpicpu0 at acpi0 acpicpu1 at acpi0 acpicpu2 at acpi0 acpicpu3 at acpi0 acpicpu4 at acpi0 acpicpu5 at acpi0 R620 bios settings Processor Settings: Logical Processor - Disabled Alternate RTID Settings - Disabled Virtualization Technology - Disabled Adjacent Cache Line Prefetch - Enabled Hardware Prefetcher - Enabled DCU Streamer Prefetcher - Enabled DCU IP Prefetcher - Enabled Logical Processor Idling - Disabled Dell Controlled Turbo - Enabled System Profile Settings: System Profile - Custom CPU Power Management - Maximum Performance Memory Frequency - Maximum Performance Turbo Boost - Enabled C1E - Disabled C states - Disabled Monitor/Mwait - Disabled Memory Patrol Scrub - Standard Memory Refresh Rate - 1x Memory Operating Voltage - Auto Collaborative CPU Performance Control - Disabled Full dmesg and acpidump from R620 http://kosjenka.srce.hr/~hrvoje/R620_custom_dmesg.txt http://kosjenka.srce.hr/~hrvoje/R620_custom.tgz
Re: Dell R630 high interrupts on acpi0
> Date: Tue, 16 Dec 2014 15:16:58 +1000
> From: Jonathan Matthew
>
> On Sun, Dec 14, 2014 at 06:22:37PM +0100, Hrvoje Popovski wrote:
> > Hi all,
> >
> > I have got two new Dell R630 and have current on them from Sun Dec
> > 14 15:07:17. Installation went great and very fast.
> > The problem is that I see around 11k interrupts on acpi0. First I
> > thought that problem is similar to this thread
> > http://marc.info/?l=openbsd-misc&m=140551906923931&w=2
> >
> > But if in dell bios system profile settings is set to performance or
> > to DAPC there are always interrupts on acpi0.
> > In links bellow you can find acpidump and dmesg from performance and
> > DAPC settings in dell bios.
>
> We just got some r630s too, so I spent some time last week figuring out what's
> going on here. Something in the AML wants to talk to the intel MEI device.
> Normally this works, but on the new generation of dell machines (we've seen it
> on r630s and r730s), it's been moved outside the pci memory range we currently
> allow on amd64. You can see this in your dmesgs:
>
> 0:22:0: mem address conflict 0x3303000/0x10
> 0:22:1: mem address conflict 0x3302000/0x10
>
> The interrupt will keep triggering until it manages to talk to the device,
> which will never happen.
>
> kettenis@ says we can get the pci memory range information we need to deal
> with
> this from acpi. Until that happens, expanding the allowed pci memory range
> makes things work properly.
>
> ok?
ok kettenis@ (although I'd prefer if you did a s/acpi/ACPI/ in the comment).
> Index: pci_machdep.c
> ===
> RCS file: /cvs/src/sys/arch/amd64/pci/pci_machdep.c,v
> retrieving revision 1.59
> diff -u -p -u -p -r1.59 pci_machdep.c
> --- pci_machdep.c 19 Apr 2014 11:53:42 - 1.59
> +++ pci_machdep.c 16 Dec 2014 04:21:53 -
> @@ -622,13 +622,17 @@ pci_init_extents(void)
>* here. As long as vendors continue to support
>* 32-bit operating systems, we should never see BARs
>* outside that region.
> + *
> + * Dell 13G servers have important devices outside the
> + * 36-bit address space. Until we can extract the address
> + * ranges from acpi, expand the allowed range to suit.
>*/
> pcimem_ex = extent_create("pcimem", 0, 0xUL,
> M_DEVBUF, NULL, 0, EX_NOWAIT);
> if (pcimem_ex == NULL)
> return;
> - extent_alloc_region(pcimem_ex, 0x10UL,
> - 0xfff0UL, EX_NOWAIT);
> + extent_alloc_region(pcimem_ex, 0x400UL,
> + 0xfc00UL, EX_NOWAIT);
>
> for (bmp = bios_memmap; bmp->type != BIOS_MAP_END; bmp++) {
> /*
Re: Dell R630 high interrupts on acpi0
On 16.12.2014. 6:16, Jonathan Matthew wrote:
On Sun, Dec 14, 2014 at 06:22:37PM +0100, Hrvoje Popovski wrote:
Hi all,
I have got two new Dell R630 and have current on them from Sun Dec
14 15:07:17. Installation went great and very fast.
The problem is that I see around 11k interrupts on acpi0. First I
thought that problem is similar to this thread
http://marc.info/?l=openbsd-misc&m=140551906923931&w=2
But if in dell bios system profile settings is set to performance or
to DAPC there are always interrupts on acpi0.
In links bellow you can find acpidump and dmesg from performance and
DAPC settings in dell bios.
We just got some r630s too, so I spent some time last week figuring out what's
going on here. Something in the AML wants to talk to the intel MEI device.
Normally this works, but on the new generation of dell machines (we've seen it
on r630s and r730s), it's been moved outside the pci memory range we currently
allow on amd64. You can see this in your dmesgs:
0:22:0: mem address conflict 0x3303000/0x10
0:22:1: mem address conflict 0x3302000/0x10
The interrupt will keep triggering until it manages to talk to the device,
which will never happen.
kettenis@ says we can get the pci memory range information we need to deal with
this from acpi. Until that happens, expanding the allowed pci memory range
makes things work properly.
ok?
Index: pci_machdep.c
===
RCS file: /cvs/src/sys/arch/amd64/pci/pci_machdep.c,v
retrieving revision 1.59
diff -u -p -u -p -r1.59 pci_machdep.c
--- pci_machdep.c 19 Apr 2014 11:53:42 - 1.59
+++ pci_machdep.c 16 Dec 2014 04:21:53 -
@@ -622,13 +622,17 @@ pci_init_extents(void)
* here. As long as vendors continue to support
* 32-bit operating systems, we should never see BARs
* outside that region.
+*
+* Dell 13G servers have important devices outside the
+* 36-bit address space. Until we can extract the address
+* ranges from acpi, expand the allowed range to suit.
*/
pcimem_ex = extent_create("pcimem", 0, 0xUL,
M_DEVBUF, NULL, 0, EX_NOWAIT);
if (pcimem_ex == NULL)
return;
- extent_alloc_region(pcimem_ex, 0x10UL,
- 0xfff0UL, EX_NOWAIT);
+ extent_alloc_region(pcimem_ex, 0x400UL,
+ 0xfc00UL, EX_NOWAIT);
for (bmp = bios_memmap; bmp->type != BIOS_MAP_END; bmp++) {
/*
Hi,
you patch makes acpi0 calm as overeaten grandad :)
Thank you.
vmstat without your patch
# vmstat -i
interrupt total rate
irq0/clock 103588029 799
irq0/ipi 1025960
irq144/acpi0 157993005512201
irq96/mfii0 69380
irq114/em0 2702502
irq99/ehci0 1310
irq99/ehci1280
Total 168389802713004
vmstat with your patch
# vmstat -i
interrupt total rate
irq0/clock 848873 800
irq0/ipi14085 13
irq144/acpi030
irq96/mfii0 22352
irq114/em0 36853
irq99/ehci0560
irq99/ehci1280
Total 868965 819
Re: Dell R630 high interrupts on acpi0
On Sun, Dec 14, 2014 at 06:22:37PM +0100, Hrvoje Popovski wrote:
> Hi all,
>
> I have got two new Dell R630 and have current on them from Sun Dec
> 14 15:07:17. Installation went great and very fast.
> The problem is that I see around 11k interrupts on acpi0. First I
> thought that problem is similar to this thread
> http://marc.info/?l=openbsd-misc&m=140551906923931&w=2
>
> But if in dell bios system profile settings is set to performance or
> to DAPC there are always interrupts on acpi0.
> In links bellow you can find acpidump and dmesg from performance and
> DAPC settings in dell bios.
We just got some r630s too, so I spent some time last week figuring out what's
going on here. Something in the AML wants to talk to the intel MEI device.
Normally this works, but on the new generation of dell machines (we've seen it
on r630s and r730s), it's been moved outside the pci memory range we currently
allow on amd64. You can see this in your dmesgs:
0:22:0: mem address conflict 0x3303000/0x10
0:22:1: mem address conflict 0x3302000/0x10
The interrupt will keep triggering until it manages to talk to the device,
which will never happen.
kettenis@ says we can get the pci memory range information we need to deal with
this from acpi. Until that happens, expanding the allowed pci memory range
makes things work properly.
ok?
Index: pci_machdep.c
===
RCS file: /cvs/src/sys/arch/amd64/pci/pci_machdep.c,v
retrieving revision 1.59
diff -u -p -u -p -r1.59 pci_machdep.c
--- pci_machdep.c 19 Apr 2014 11:53:42 - 1.59
+++ pci_machdep.c 16 Dec 2014 04:21:53 -
@@ -622,13 +622,17 @@ pci_init_extents(void)
* here. As long as vendors continue to support
* 32-bit operating systems, we should never see BARs
* outside that region.
+*
+* Dell 13G servers have important devices outside the
+* 36-bit address space. Until we can extract the address
+* ranges from acpi, expand the allowed range to suit.
*/
pcimem_ex = extent_create("pcimem", 0, 0xUL,
M_DEVBUF, NULL, 0, EX_NOWAIT);
if (pcimem_ex == NULL)
return;
- extent_alloc_region(pcimem_ex, 0x10UL,
- 0xfff0UL, EX_NOWAIT);
+ extent_alloc_region(pcimem_ex, 0x400UL,
+ 0xfc00UL, EX_NOWAIT);
for (bmp = bios_memmap; bmp->type != BIOS_MAP_END; bmp++) {
/*
Re: Dell R630 high interrupts on acpi0
> On 16 Dec 2014, at 15:16, Jonathan Matthew wrote:
>
> On Sun, Dec 14, 2014 at 06:22:37PM +0100, Hrvoje Popovski wrote:
>> Hi all,
>>
>> I have got two new Dell R630 and have current on them from Sun Dec
>> 14 15:07:17. Installation went great and very fast.
>> The problem is that I see around 11k interrupts on acpi0. First I
>> thought that problem is similar to this thread
>> http://marc.info/?l=openbsd-misc&m=140551906923931&w=2
>>
>> But if in dell bios system profile settings is set to performance or
>> to DAPC there are always interrupts on acpi0.
>> In links bellow you can find acpidump and dmesg from performance and
>> DAPC settings in dell bios.
>
> We just got some r630s too, so I spent some time last week figuring out what's
> going on here. Something in the AML wants to talk to the intel MEI device.
> Normally this works, but on the new generation of dell machines (we've seen it
> on r630s and r730s), it's been moved outside the pci memory range we currently
> allow on amd64. You can see this in your dmesgs:
>
> 0:22:0: mem address conflict 0x3303000/0x10
> 0:22:1: mem address conflict 0x3302000/0x10
>
> The interrupt will keep triggering until it manages to talk to the device,
> which will never happen.
we've also experienced this on r720s configured in Performance mode in the BIOS
too. others have hit this on r620s as well (look for "Dell R620 high ACPI
Interrupt rate" on misc@). the diff below fixes them too.
dlg
>
> kettenis@ says we can get the pci memory range information we need to deal
> with
> this from acpi. Until that happens, expanding the allowed pci memory range
> makes things work properly.
>
> ok?
>
>
> Index: pci_machdep.c
> ===
> RCS file: /cvs/src/sys/arch/amd64/pci/pci_machdep.c,v
> retrieving revision 1.59
> diff -u -p -u -p -r1.59 pci_machdep.c
> --- pci_machdep.c 19 Apr 2014 11:53:42 - 1.59
> +++ pci_machdep.c 16 Dec 2014 04:21:53 -
> @@ -622,13 +622,17 @@ pci_init_extents(void)
>* here. As long as vendors continue to support
>* 32-bit operating systems, we should never see BARs
>* outside that region.
> + *
> + * Dell 13G servers have important devices outside the
> + * 36-bit address space. Until we can extract the address
> + * ranges from acpi, expand the allowed range to suit.
>*/
> pcimem_ex = extent_create("pcimem", 0, 0xUL,
> M_DEVBUF, NULL, 0, EX_NOWAIT);
> if (pcimem_ex == NULL)
> return;
> - extent_alloc_region(pcimem_ex, 0x10UL,
> - 0xfff0UL, EX_NOWAIT);
> + extent_alloc_region(pcimem_ex, 0x400UL,
> + 0xfc00UL, EX_NOWAIT);
>
> for (bmp = bios_memmap; bmp->type != BIOS_MAP_END; bmp++) {
> /*
Dell R630 high interrupts on acpi0
Hi all, I have got two new Dell R630 and have current on them from Sun Dec 14 15:07:17. Installation went great and very fast. The problem is that I see around 11k interrupts on acpi0. First I thought that problem is similar to this thread http://marc.info/?l=openbsd-misc&m=140551906923931&w=2 But if in dell bios system profile settings is set to performance or to DAPC there are always interrupts on acpi0. In links bellow you can find acpidump and dmesg from performance and DAPC settings in dell bios. http://kosjenka.srce.hr/~hrvoje/R630_acpidump_performance.tar.gz http://kosjenka.srce.hr/~hrvoje/dmesg_performance.txt http://kosjenka.srce.hr/~hrvoje/R630_acpidump_dapc.tgz http://kosjenka.srce.hr/~hrvoje/dmesg_dapc.txt
