repeated panics with ix
I keep getting panics on this machine in ixgbe routines after a few days of light traffic. I only have IP-KVM access which attaches as a USB keyboard, so machdep.forceukbd doesn't work once panicked, so I can't do anything in DDB. uvm_fault(0xfd8876955228, 0x40, 0, 1) -> e fatal page fault in supervisor mode trap type 6 code 0 rip 8167e883 cs 8 rflags 10206 cr2 40 cpl 0 rsp 8000217fde30 gsbase 0x800021368ff0 kgsbase 0x0 panic: trap type 6, code=0, pc=8167e883 Starting stack trace... panic() at panic+0x11b kerntrap(8000217fdd80) at kerntrap+0x114 alltraps_kern(6,0,8023d000,0,8023d2b0,8022cf00) at alltraps_kern+0x7b ixgbe_start(8023d2b0) at ixgbe_start+0x93 ifq_serialize(8023d2b0,8023d350) at ifq_serialize+0xf3 if_enqueue_ifq(8023d048,fd807cb98600) at if_enqueue_ifq+0x72 ether_output(8023d048,fd807cb98600,80218e90,fd877fc87540) at ether_output+0x7e ip_output(fd807cb98600,0,fd887315ca48,800,0,fd887315c9d8) at ip_output+0x98a tcp_output(807e1aa0) at tcp_output+0x1617 tcp_usrreq(fd877903d7d8,9,fd807b8d5a00,0,0,80002180c040) at tcp_usrreq+0xff sosend(fd877903d7d8,0,8000217fe3e8,0,0,80) at sosend+0x383 dofilewritev(80002180c040,4,8000217fe3e8,0,8000217fe4e0) at dofilewritev+0xf2 sys_write(80002180c040,8000217fe470,8000217fe4e0) at sys_write+0x54 syscall(8000217fe550) at syscall+0x305 Xsyscall(6,4,e130ff15efe,4,4,e1577a4fc00) at Xsyscall+0x128 end of kernel end trace frame: 0x7f7eceb0, count: 242 End of stack trace. (hand-copied) uvm_fault(0x81d34360, 0x10e, 0, 1) -> e fatal page fault in supervisor mode trap type 6 code 0 rip 8191f405 cs 8 rflags 10246 cr2 10e cpl 7 rsp 800021554ad0 gsbase 0x81c97ff0 kgsbase 0x0 panic: trap type 6, code=0, pc=8191f405 Starting stack trace... panic() at panic+0x11b kerntrap(800021554a20) at kerntrap+0x114 alltraps_kern(6,0,0,0,8021ecc0,8023d000) at alltraps_kern+0x7b ixgbe_txeof(8022cf00) at ixgbe_txeof+0x25 ixgbe_intr(8023d000) at ixgbe_intr+0x6f intr_handler(800021554be0,80224880) at intr_handler+0x6e Xintr_ioapic_edge17_untramp(0,0,1388,0,0,f81c986f8) at Xintr_ioapic_edge17_untramp+0x19f acpicpu_idle() at acpicpu_idle+0x1db sched_idle(f81c97ff0) at sched_idle+0x205 end trace frame: 0x0, count: 248 End of stack trace. OpenBSD 6.5 (GENERIC.MP) #1: Mon May 27 18:27:59 CEST 2019 r...@syspatch-65-amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 34156396544 (32574MB) avail mem = 33111588864 (31577MB) mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 3.0 @ 0x7fb76000 (32 entries) bios0: vendor American Megatrends Inc. version "2.2" date 03/09/2019 bios0: Supermicro Super Server acpi0 at bios0: rev 2 acpi0: sleep states S0 S4 S5 acpi0: tables DSDT FACP APIC FPDT FIDT SPMI MCFG HPET LPIT SSDT SSDT SSDT DBGP DBG2 SSDT PRAD SSDT UEFI SSDT DMAR EINJ ERST BERT HEST acpi0: wakeup devices PEGP(S4) PEG0(S4) PEGP(S4) PEG1(S4) PEG2(S4) PXSX(S4) RP09(S4) PXSX(S4) RP10(S4) PXSX(S4) RP11(S4) PXSX(S4) RP12(S4) PXSX(S4) RP13(S4) PXSX(S4) [...] acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Xeon(R) CPU E3-1270 v6 @ 3.80GHz, 3801.12 MHz, 06-9e-09 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN cpu0: 256KB 64b/line 8-way L2 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges cpu0: apic clock running at 23MHz cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4.1, IBE cpu1 at mainbus0: apid 2 (application processor) cpu1: Intel(R) Xeon(R) CPU E3-1270 v6 @ 3.80GHz, 3800.00 MHz, 06-9e-09 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN cpu1: 256KB 64b/line 8-way L2 cache cpu1: smt 0, core 1, package 0 cpu2 at mainbus0: apid 4 (application processor) cpu2: Intel(R) Xeon(R) CPU E3-1270 v6 @ 3.80GHz, 3800.00
Re: ifiq_input pressure drop too aggressive for msk interfaces
On Mon, Jul 29, 2019 at 12:22:28AM +0200, Olivier Ta??bi wrote:
> On one of my machines equipped with two msk(4) gigabit ethernet devices,
> the reintroduction (rev. 1.32) of counting backpressure in ifiq_input() in
> sys/net/ifq.c causes drops when more than 8 ethernet frames arrive in a short
> amount of time. I noticed this because this machine is an NFS server, mounted
> with the (client-side) option "-w 32768", but the drops are reproducible
> (visible with netstat -I msk*) simply by sending to the machine a large (e.g.
> 17kB) UDP packet which gets fragmented into >8 IP packets.
> Increasing net.link.ifrxq.pressure_drop (e.g. from the default value 8 to 40)
> suppresses this problem, although I am not sure whether this is the correct
> solution.
i think i see the more fundamental problem. the diff below should fix it
so you don't need to tune the ifrxq pressure levels.
this basically tweaks msk so it enqueues packets for the stack once per
interrupt, rather than once per rx completion event. this helps when you
rx multiple packets per interrupt like you are when a packet gets
fragmented. while here it uses ifiq_input so it can apply earlier
backpressure.
i'll commit this soon, so hopefully it will be in the tree for you to
test.
Index: if_msk.c
===
RCS file: /cvs/src/sys/dev/pci/if_msk.c,v
retrieving revision 1.131
diff -u -p -r1.131 if_msk.c
--- if_msk.c6 Jan 2018 03:11:04 - 1.131
+++ if_msk.c30 Jul 2019 03:59:11 -
@@ -134,7 +134,7 @@ int mskcprint(void *, const char *);
int msk_intr(void *);
void msk_intr_yukon(struct sk_if_softc *);
static inline int msk_rxvalid(struct sk_softc *, u_int32_t, u_int32_t);
-void msk_rxeof(struct sk_if_softc *, uint16_t, uint32_t);
+void msk_rxeof(struct sk_if_softc *, struct mbuf_list *, uint16_t, uint32_t);
void msk_txeof(struct sk_if_softc *);
static unsigned int msk_encap(struct sk_if_softc *, struct mbuf *, uint32_t);
void msk_start(struct ifnet *);
@@ -1591,11 +1591,11 @@ msk_rxvalid(struct sk_softc *sc, u_int32
}
void
-msk_rxeof(struct sk_if_softc *sc_if, uint16_t len, uint32_t rxstat)
+msk_rxeof(struct sk_if_softc *sc_if, struct mbuf_list *ml,
+uint16_t len, uint32_t rxstat)
{
struct sk_softc *sc = sc_if->sk_softc;
struct ifnet*ifp = _if->arpcom.ac_if;
- struct mbuf_listml = MBUF_LIST_INITIALIZER();
struct mbuf *m = NULL;
int prod, cons, tail;
bus_dmamap_tmap;
@@ -1640,8 +1640,7 @@ msk_rxeof(struct sk_if_softc *sc_if, uin
m->m_pkthdr.len = m->m_len = len;
- ml_enqueue(, m);
- if_input(ifp, );
+ ml_enqueue(ml, m);
}
void
@@ -1770,8 +1769,12 @@ msk_intr(void *xsc)
struct sk_if_softc *sc_if;
struct sk_if_softc *sc_if0 = sc->sk_if[SK_PORT_A];
struct sk_if_softc *sc_if1 = sc->sk_if[SK_PORT_B];
+ struct mbuf_listml[2] = {
+ MBUF_LIST_INITIALIZER(),
+ MBUF_LIST_INITIALIZER(),
+ };
struct ifnet*ifp0 = NULL, *ifp1 = NULL;
- int claimed = 0, rx[2] = {0, 0};
+ int claimed = 0;
u_int32_t status;
struct msk_status_desc *cur_st;
@@ -1809,8 +1812,8 @@ msk_intr(void *xsc)
switch (cur_st->sk_opcode) {
case SK_Y2_STOPC_RXSTAT:
sc_if = sc->sk_if[cur_st->sk_link & 0x01];
- rx[cur_st->sk_link & 0x01] = 1;
- msk_rxeof(sc_if, lemtoh16(_st->sk_len),
+ msk_rxeof(sc_if, [cur_st->sk_link & 0x01],
+ lemtoh16(_st->sk_len),
lemtoh32(_st->sk_status));
break;
case SK_Y2_STOPC_TXSTAT:
@@ -1837,12 +1840,16 @@ msk_intr(void *xsc)
CSR_WRITE_4(sc, SK_Y2_ICR, 2);
- if (rx[0]) {
+ if (!ml_empty([0])) {
+ if (ifiq_input(>if_rcv, [0]))
+ if_rxr_livelocked(_if0->sk_cdata.sk_rx_ring);
msk_fill_rx_ring(sc_if0);
SK_IF_WRITE_2(sc_if0, 0, SK_RXQ1_Y2_PREF_PUTIDX,
sc_if0->sk_cdata.sk_rx_prod);
}
- if (rx[1]) {
+ if (!ml_empty([1])) {
+ if (ifiq_input(>if_rcv, [1]))
+ if_rxr_livelocked(_if1->sk_cdata.sk_rx_ring);
msk_fill_rx_ring(sc_if1);
SK_IF_WRITE_2(sc_if1, 0, SK_RXQ1_Y2_PREF_PUTIDX,
sc_if1->sk_cdata.sk_rx_prod);
Re: printf(1) man page has a small omission
I have a diff attached which fixes the man page. On Mon, Jun 3, 2019 at 2:12 PM Andras Farkas wrote: > > https://man.openbsd.org/man1/printf.1 > The section on the b format (%b) neglects to mention that for that > format, it's \0num rather than \num > Because of the way OpenBSD's printf is made, both \0num and \num work > for %b, but, \0num is more correct and portable when using printf(1)'s > %b format. > https://pubs.opengroup.org/onlinepubs/9699919799/utilities/printf.html > > For comparison: > https://www.freebsd.org/cgi/man.cgi?query=printf=0=1=FreeBSD+12.0-RELEASE+and+Ports=default=html > https://netbsd.gw.com/cgi-bin/man-cgi?printf+1 Index: printf.1 === RCS file: /cvs/src/usr.bin/printf/printf.1,v retrieving revision 1.32 diff -u -p -r1.32 printf.1 --- printf.12 Jun 2019 06:16:37 - 1.32 +++ printf.130 Jul 2019 03:51:32 - @@ -328,6 +328,13 @@ all remaining characters in this argumen and all remaining characters in the .Ar format string. +The string +.Ar argument +uses +.Cm \e0 Ns Ar num +rather than +.Cm \e Ns Ar num +for octal escapes. .It Cm c The first character of .Ar argument
ifiq_input pressure drop too aggressive for msk interfaces
On one of my machines equipped with two msk(4) gigabit ethernet devices, the reintroduction (rev. 1.32) of counting backpressure in ifiq_input() in sys/net/ifq.c causes drops when more than 8 ethernet frames arrive in a short amount of time. I noticed this because this machine is an NFS server, mounted with the (client-side) option "-w 32768", but the drops are reproducible (visible with netstat -I msk*) simply by sending to the machine a large (e.g. 17kB) UDP packet which gets fragmented into >8 IP packets. Increasing net.link.ifrxq.pressure_drop (e.g. from the default value 8 to 40) suppresses this problem, although I am not sure whether this is the correct solution. OpenBSD 6.5-current (GENERIC.MP) #21: Fri Jul 26 13:25:41 CEST 2019 r...@gauss.lan:/sys/arch/amd64/compile/GENERIC.MP real mem = 4269277184 (4071MB) avail mem = 4129763328 (3938MB) mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.5 @ 0xf0700 (81 entries) bios0: vendor American Megatrends Inc. version "0404" date 01/28/2010 bios0: ASUSTeK Computer INC. P6TD DELUXE acpi0 at bios0: ACPI 3.0 acpi0: sleep states S0 S1 S3 S4 S5 acpi0: tables DSDT FACP APIC MCFG OEMB HPET OSFR SSDT acpi0: wakeup devices NPE2(S4) NPE4(S4) NPE5(S4) NPE6(S4) NPE8(S4) NPE9(S4) NPEA(S4) P0P1(S4) USB0(S4) USB1(S4) USB2(S4) USB5(S4) EUSB(S4) USB3(S4) USB4(S4) USB6(S4) [...] acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz, 3207.67 MHz, 06-1a-04 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,MELTDOWN cpu0: 256KB 64b/line 8-way L2 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges cpu0: apic clock running at 133MHz cpu0: mwait min=64, max=64, C-substates=0.2.1.1, IBE cpu1 at mainbus0: apid 2 (application processor) cpu1: Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz, 2674.42 MHz, 06-1a-04 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,MELTDOWN cpu1: 256KB 64b/line 8-way L2 cache cpu1: smt 0, core 1, package 0 cpu2 at mainbus0: apid 4 (application processor) cpu2: Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz, 2672.74 MHz, 06-1a-04 cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,MELTDOWN cpu2: 256KB 64b/line 8-way L2 cache cpu2: smt 0, core 2, package 0 cpu3 at mainbus0: apid 6 (application processor) cpu3: Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz, 2672.74 MHz, 06-1a-04 cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,MELTDOWN cpu3: 256KB 64b/line 8-way L2 cache cpu3: smt 0, core 3, package 0 cpu4 at mainbus0: apid 1 (application processor) cpu4: Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz, 2672.73 MHz, 06-1a-04 cpu4: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,MELTDOWN cpu4: 256KB 64b/line 8-way L2 cache cpu4: smt 1, core 0, package 0 cpu5 at mainbus0: apid 3 (application processor) cpu5: Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz, 2672.73 MHz, 06-1a-04 cpu5: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,MELTDOWN cpu5: 256KB 64b/line 8-way L2 cache cpu5: smt 1, core 1, package 0 cpu6 at mainbus0: apid 5 (application processor) cpu6: Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz, 2672.73 MHz, 06-1a-04 cpu6: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,MELTDOWN cpu6: 256KB 64b/line 8-way L2 cache cpu6: smt 1, core 2, package 0 cpu7 at mainbus0: apid 7 (application processor) cpu7: Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz, 2672.73 MHz, 06-1a-04 cpu7:
