Hi,
One of the problems obstructing my dynamic clock interrupt patch is
that clock interrupts on powerpc don't (can't?) behave the same as
clock interrupts on amd64, arm64, and sparc64.
In particular, for historical reasons, on powerpc you cannot mask
decrementer (DEC) interrupts without *also* masking other interrupts
that we need to (generally) leave unmasked.
The upshot is that the DEC is unmasked at IPL_CLOCK and IPL_HIGH on
powerpc. It's always running, it can arrive at any time. We work
around the obvious problem this poses by postponing clock interrupt
work to a later tick if a DEC interrupt arrives when the CPU is at
IPL_CLOCK or IPL_HIGH.
This solution is insufficient for a machine-independent clock
interrupt subsystem like the one in my patch.
The only way forward I can see is to instead postpone clock interrupt
work until the next splx(9) call wherein the CPU's IPL is dropping
below IPL_CLOCK.
This patch does that. We need to raise the DEC exception immediately
after we change the IPL, so there is a little bit of code duplicated
across the various splx(9) implementations for powerpc and macppc.
The changes in macppc/clock.c are hopefully straightforward.
This boots on my PowerMac G5 ("PowerMac7,3") and has survived two
`make build` runs and several more kernel builds.
Caveat: this machine is not quite stable. I can't run parallel
userland builds, i.e. `make -j2 build` without eventually hanging the
machine. Also, sometimes it hangs at boot while /etc/netstart is
running. These problems existed before this patch and remain after
the patch is applied.
However, this patch does not seem to have made the machine *more*
unstable, which is a good sign, I think.
My machine uses openpic(4). I would appreciate tests on a
non-openpic(4) Mac, though all tests are appreciated.
Thoughts? Tweaks?
If we merge this change I plan make an equivalent change on powerpc64.
dmesg below, patch attached at the end.
-Scott
[ using 1321372 bytes of bsd ELF symbol table ]
console out [ATY,Whelk_A] console in [keyboard], using USB
using parent ATY,WhelkParent:: memaddr a0000000, size 10000000 : consaddr
a0008000 : ioaddr 90020000, size 20000: width 1152 linebytes 1280 height 870
depth 8
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
Copyright (c) 1995-2022 OpenBSD. All rights reserved. https://www.OpenBSD.org
OpenBSD 7.1-current (GENERIC.MP) #4: Thu Jun 23 10:55:15 CDT 2022
ssc@peanut.local:/usr/src/sys/arch/macppc/compile/GENERIC.MP
real mem = 2147483648 (2048MB)
avail mem = 2051514368 (1956MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root: model PowerMac7,3
cpu0 at mainbus0: 970FX (Revision 0x300): 1800 MHz
cpu1 at mainbus0: 970FX (Revision 0x300): 1800 MHz
mem0 at mainbus0
spdmem0 at mem0: 1GB DDR SDRAM non-parity PC3200CL3.0
spdmem1 at mem0: 1GB DDR SDRAM non-parity PC3200CL3.0
spdmem2 at mem0: 1GB DDR SDRAM non-parity PC3200CL3.0
spdmem3 at mem0: 1GB DDR SDRAM non-parity PC3200CL3.0
memc0 at mainbus0: u3 rev 0xb3
kiic0 at memc0 offset 0xf8001000
iic0 at kiic0
lmtemp0 at iic0 addr 0x4a: ds1775
maxtmp0 at iic0 addr 0x4c: max6690
maxtmp1 at iic0 addr 0x4e: max6690
"cy28508" at iic0 addr 0x69 not configured
"cy2213" at iic0 addr 0x65 not configured
fcu0 at iic0 addr 0xaf
"pca9556" at iic0 addr 0x18 not configured
adc0 at iic0 addr 0x2c: ad7417
"24256" at iic0 addr 0x50 not configured
"pca9556" at iic0 addr 0x19 not configured
adc1 at iic0 addr 0x2d: ad7417
"24256" at iic0 addr 0x51 not configured
"dart" at memc0 offset 0xf8033000 not configured
"mpic" at memc0 offset 0xf8040000 not configured
mpcpcibr0 at mainbus0 pci: u3-agp
pci0 at mpcpcibr0 bus 0
pchb0 at pci0 dev 11 function 0 "Apple U3 AGP" rev 0x00
appleagp0 at pchb0
agp0 at appleagp0: aperture at 0x0, size 0x10000000
radeondrm0 at pci0 dev 16 function 0 "ATI Radeon 9600" rev 0x00
drm0 at radeondrm0
radeondrm0: irq 48
ht0 at mainbus0: u3-ht, 6 devices
pci1 at ht0 bus 0
hpb0 at pci1 dev 1 function 0 "Apple U3" rev 0x00: 85 sources
pci2 at hpb0 bus 1
macobio0 at pci2 dev 7 function 0 "Apple K2 Macio" rev 0x60
openpic0 at macobio0 offset 0x40000: version 0x4614 feature 770302 LE
macgpio0 at macobio0 offset 0x50
"pmu-interrupt" at macgpio0 offset 0x9 not configured
"programmer-switch" at macgpio0 offset 0x11 not configured
"modem-reset" at macgpio0 offset 0x1d not configured
"modem-power" at macgpio0 offset 0x1e not configured
"fcu-interrupt" at macgpio0 offset 0x15 not configured
"fcu-hw-reset" at macgpio0 offset 0x3a not configured
"slewing-done" at macgpio0 offset 0x23 not configured
"codec-input-data-mux" at macgpio0 offset 0xb not configured
"line-input-detect" at macgpio0 offset 0xc not configured
"codec-error-irq" at macgpio0 offset 0xd not configured
"dig-hw-reset" at macgpio0 offset 0x14 not configured
"line-output-detect" at macgpio0 offset 0x16 not configured
"headphone-detect" at macgpio0 offset 0x17 not configured
"codec-irq" at macgpio0 offset 0x18 not configured
"headphone-mute" at macgpio0 offset 0x1f not configured
"amp-mute" at macgpio0 offset 0x20 not configured
"hw-reset" at macgpio0 offset 0x24 not configured
"line-output-mute" at macgpio0 offset 0x25 not configured
"codec-clock-mux" at macgpio0 offset 0x26 not configured
"escc-legacy" at macobio0 offset 0x12000 not configured
zs0 at macobio0 offset 0x13000: irq 22,23
zstty0 at zs0 channel 0
zstty1 at zs0 channel 1
kiic1 at macobio0 offset 0x18000
iic1 at kiic1
aoa0 at macobio0 offset 0x10000: irq 30,1,2
"timer" at macobio0 offset 0x15000 not configured
adb0 at macobio0 offset 0x16000
apm0 at adb0: battery flags 0x9, 0% charged
piic0 at adb0
iic2 at piic0
"fans" at macobio0 offset 0x4c not configured
audio0 at aoa0
ohci0 at pci2 dev 8 function 0 "Apple K2 USB" rev 0x00: irq 27, version 1.0,
legacy support
ohci1 at pci2 dev 9 function 0 "Apple K2 USB" rev 0x00: irq 28, version 1.0,
legacy support
usb0 at ohci0: USB revision 1.0
uhub0 at usb0 configuration 1 interface 0 "Apple OHCI root hub" rev 1.00/1.00
addr 1
usb1 at ohci1: USB revision 1.0
uhub1 at usb1 configuration 1 interface 0 "Apple OHCI root hub" rev 1.00/1.00
addr 1
ppb0 at pci1 dev 2 function 0 "Apple U3" rev 0x00
pci3 at ppb0 bus 5
bwi0 at pci3 dev 1 function 0 "Broadcom BCM4306" rev 0x03: irq 57, address
00:11:24:8e:99:28
ohci2 at pci3 dev 11 function 0 "NEC USB" rev 0x43: irq 63, version 1.0
ohci3 at pci3 dev 11 function 1 "NEC USB" rev 0x43: irq 63, version 1.0
ehci0 at pci3 dev 11 function 2 "NEC USB" rev 0x04: irq 63
usb2 at ehci0: USB revision 2.0
uhub2 at usb2 configuration 1 interface 0 "NEC EHCI root hub" rev 2.00/1.00
addr 1
usb3 at ohci2: USB revision 1.0
uhub3 at usb3 configuration 1 interface 0 "NEC OHCI root hub" rev 1.00/1.00
addr 1
usb4 at ohci3: USB revision 1.0
uhub4 at usb4 configuration 1 interface 0 "NEC OHCI root hub" rev 1.00/1.00
addr 1
ppb1 at pci1 dev 3 function 0 "Apple U3" rev 0x00
pci4 at ppb1 bus 2
kauaiata0 at pci4 dev 13 function 0 "Apple K2 ATA" rev 0x00
wdc0 at kauaiata0 irq 39: DMA
atapiscsi0 at wdc0 channel 0 drive 0
scsibus1 at atapiscsi0: 2 targets
cd0 at scsibus1 targ 0 lun 0: <PIONEER, DVD-RW DVR-117D, B705> removable
cd0(wdc0:0:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 4
"Apple K2 Firewire" rev 0x00 at pci4 dev 14 function 0 not configured
ppb2 at pci1 dev 4 function 0 "Apple U3" rev 0x00
pci5 at ppb2 bus 3
gem0 at pci5 dev 15 function 0 "Apple K2 GMAC" rev 0x00: irq 41, address
00:0d:93:5e:0b:dc
brgphy0 at gem0 phy 1: BCM54K2 10/100/1000baseT PHY, rev. 0
ppb3 at pci1 dev 5 function 0 "Apple U3" rev 0x00
pci6 at ppb3 bus 4
pciide0 at pci6 dev 12 function 0 "ServerWorks K2 SATA" rev 0x00: DMA
pciide0: using irq 0 for native-PCI interrupt
pciide0: port 0: PHY offline
pciide0: port 1: 1.5Gb/s
wd0 at pciide0 channel 1 drive 0: <WDC WD7501AALS-00J7B0>
wd0: 16-sector PIO, LBA48, 715403MB, 1465147055 sectors
wd0(pciide0:1:0): using PIO mode 4, Ultra-DMA mode 6
pciide0: port 2: PHY offline
pciide0: port 3: PHY offline
uhidev0 at uhub3 port 1 configuration 1 interface 0 "CHICONY USB NetVista Full
Width Keyboard" rev 1.10/1.02 addr 2
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 variable keys, 6 key codes
wskbd0 at ukbd0: console keyboard
uhidev1 at uhub0 port 1 configuration 1 interface 0 "Apple Computer HID-proxy"
rev 1.10/12.41 addr 2
uhidev1: iclass 3/1
ukbd1 at uhidev1: 8 variable keys, 6 key codes
wskbd1 at ukbd1 mux 1
uhidev2 at uhub0 port 1 configuration 1 interface 1 "Apple Computer HID-proxy"
rev 1.10/12.41 addr 2
uhidev2: iclass 3/1
ums0 at uhidev2: 5 buttons
wsmouse0 at ums0 mux 0
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
bootpath: /ht/pci@5/k2-sata-root/k2-sata@1/disk@0:/bsd
root on wd0a (f3171bf68f5b0fa0.a) swap on wd0b dump on wd0b
radeondrm0: RV350
radeondrm0: 1600x1200, 32bpp
wsdisplay0 at radeondrm0 mux 1: console (std, vt100 emulation), using wskbd0
wskbd1: connecting to wsdisplay0
wsdisplay0: screen 1-5 added (std, vt100 emulation)
Bogus possible_clones: [ENCODER:45:TMDS-45] possible_clones=0x6 (full encoder
mask=0x7)
Bogus possible_clones: [ENCODER:46:TV-46] possible_clones=0x5 (full encoder
mask=0x7)
Bogus possible_clones: [ENCODER:48:DAC-48] possible_clones=0x3 (full encoder
mask=0x7)
Index: macppc/macppc/clock.c
===================================================================
RCS file: /cvs/src/sys/arch/macppc/macppc/clock.c,v
retrieving revision 1.48
diff -u -p -r1.48 clock.c
--- macppc/macppc/clock.c 23 Feb 2021 04:44:30 -0000 1.48
+++ macppc/macppc/clock.c 24 Jun 2022 02:49:58 -0000
@@ -128,6 +128,20 @@ decr_intr(struct clockframe *frame)
return;
/*
+ * We can't actually mask DEC interrupts, i.e. mask MSR(EE),
+ * at or above IPL_CLOCK without masking other essential
+ * interrupts. To simulate masking, we retrigger the DEC
+ * by hand from splx(9) the next time our IPL drops below
+ * IPL_CLOCK.
+ */
+ if (ci->ci_cpl >= IPL_CLOCK) {
+ ci->ci_dec_deferred = 1;
+ ppc_mtdec(UINT32_MAX >> 1); /* clear DEC exception */
+ return;
+ }
+ ci->ci_dec_deferred = 0;
+
+ /*
* Based on the actual time delay since the last decrementer reload,
* we arrange for earlier interrupt next time.
*/
@@ -160,39 +174,35 @@ decr_intr(struct clockframe *frame)
*/
ppc_mtdec(nextevent - tb);
- if (ci->ci_cpl >= IPL_CLOCK) {
- ci->ci_statspending += nstats;
- } else {
- nstats += ci->ci_statspending;
- ci->ci_statspending = 0;
-
- s = splclock();
-
- /*
- * Reenable interrupts
- */
- ppc_intr_enable(1);
-
- /*
- * Do standard timer interrupt stuff.
- */
- while (ci->ci_lasttb < ci->ci_prevtb) {
- /* sync lasttb with hardclock */
- ci->ci_lasttb += ticks_per_intr;
- clk_count.ec_count++;
- hardclock(frame);
- }
-
- while (nstats-- > 0)
- statclock(frame);
-
- splx(s);
- (void) ppc_intr_disable();
-
- /* if a tick has occurred while dealing with these,
- * dont service it now, delay until the next tick.
- */
+ nstats += ci->ci_statspending;
+ ci->ci_statspending = 0;
+
+ s = splclock();
+
+ /*
+ * Reenable interrupts
+ */
+ ppc_intr_enable(1);
+
+ /*
+ * Do standard timer interrupt stuff.
+ */
+ while (ci->ci_lasttb < ci->ci_prevtb) {
+ /* sync lasttb with hardclock */
+ ci->ci_lasttb += ticks_per_intr;
+ clk_count.ec_count++;
+ hardclock(frame);
}
+
+ while (nstats-- > 0)
+ statclock(frame);
+
+ splx(s);
+ (void) ppc_intr_disable();
+
+ /* if a tick has occurred while dealing with these,
+ * dont service it now, delay until the next tick.
+ */
}
void cpu_startclock(void);
Index: macppc/dev/openpic.c
===================================================================
RCS file: /cvs/src/sys/arch/macppc/dev/openpic.c,v
retrieving revision 1.89
diff -u -p -r1.89 openpic.c
--- macppc/dev/openpic.c 21 Feb 2022 10:38:50 -0000 1.89
+++ macppc/dev/openpic.c 24 Jun 2022 02:49:59 -0000
@@ -382,6 +382,10 @@ openpic_splx(int newcpl)
intr = ppc_intr_disable();
openpic_setipl(newcpl);
+ if (ci->ci_dec_deferred && newcpl < IPL_CLOCK) {
+ ppc_mtdec(0);
+ ppc_mtdec(UINT32_MAX); /* raise DEC exception */
+ }
if (newcpl < IPL_SOFTTTY && (ci->ci_ipending & ppc_smask[newcpl])) {
s = splsofttty();
dosoftint(newcpl);
Index: macppc/dev/macintr.c
===================================================================
RCS file: /cvs/src/sys/arch/macppc/dev/macintr.c,v
retrieving revision 1.56
diff -u -p -r1.56 macintr.c
--- macppc/dev/macintr.c 13 Mar 2022 12:33:01 -0000 1.56
+++ macppc/dev/macintr.c 24 Jun 2022 02:49:59 -0000
@@ -170,6 +170,10 @@ macintr_splx(int newcpl)
intr = ppc_intr_disable();
macintr_setipl(newcpl);
+ if (ci->ci_dec_deferred && newcpl < IPL_CLOCK) {
+ ppc_mtdec(0);
+ ppc_mtdec(UINT32_MAX); /* raise DEC exception */
+ }
if ((newcpl < IPL_SOFTTTY && ci->ci_ipending & ppc_smask[newcpl])) {
s = splsofttty();
dosoftint(newcpl);
Index: powerpc/powerpc/intr.c
===================================================================
RCS file: /cvs/src/sys/arch/powerpc/powerpc/intr.c,v
retrieving revision 1.9
diff -u -p -r1.9 intr.c
--- powerpc/powerpc/intr.c 13 Sep 2015 14:06:40 -0000 1.9
+++ powerpc/powerpc/intr.c 24 Jun 2022 02:49:59 -0000
@@ -120,6 +120,11 @@ ppc_dflt_splx(int newcpl)
ci->ci_cpl = newcpl;
+ if (ci->ci_dec_deferred && newcpl < IPL_CLOCK) {
+ ppc_mtdec(0);
+ ppc_mtdec(UINT32_MAX); /* raise DEC exception */
+ }
+
if (ci->ci_ipending & ppc_smask[newcpl])
dosoftint(newcpl);
}
Index: powerpc/include/cpu.h
===================================================================
RCS file: /cvs/src/sys/arch/powerpc/include/cpu.h,v
retrieving revision 1.71
diff -u -p -r1.71 cpu.h
--- powerpc/include/cpu.h 10 Feb 2022 05:48:02 -0000 1.71
+++ powerpc/include/cpu.h 24 Jun 2022 02:49:59 -0000
@@ -55,6 +55,7 @@ struct cpu_info {
volatile int ci_want_resched;
volatile int ci_cpl;
volatile int ci_ipending;
+ volatile int ci_dec_deferred;
volatile int ci_flags;
#define CI_FLAGS_SLEEPING 2
