Hi,
Apparently mips64, i.e. octeon and loongson, has the same problem as
powerpc/macppc and powerpc64. The timer interrupt is normally only
logically masked, not physically masked in the hardware, when we're
running at or above IPL_CLOCK. If we arrive at cp0_int5() when the
clock interrupt is logically masked we postpone all work until the
next tick. This is a problem for my WIP clock interrupt work.
So, this patch is basically the same as what I did for macppc and what
I have proposed for powerpc64.
- Add a new member, ci_timer_deferred, to mips64's cpu_info struct.
While here, remove ci_pendingticks. We don't need it anymore.
- If we get to cp0_int5() and our IPL is too high, set
cpu_info.ci_timer_deferred and return.
- If we get to cp0_int5() and our IPL is low enough, clear
cpu_info.ci_timer_deferred and do clock interrupt work.
- In splx(9), if the new IPL is low enough and cpu_info.ci_timer_deferred
is set, trigger the clock interrupt.
The only big difference is that mips64 uses an equality comparison
when deciding whether to arm the timer interrupt, so it's really easy
to "miss" CP0.count when you're setting CP0.compare.
To address this I've written a function, cp0_raise_int5(), that spins
until it is sure the timer interrupt will go off. The function needed
a bit of new code for reading CP0.cause, which I've added to
cp0access.S. I am using an initial offset of 16 cycles based on
experimentation with the machine I have access to, a 500Mhz CN50xx.
Values lower than 16 require more than one loop to arm the timer. If
that value is insufficient for other machines we can try passing the
initial offset as an argument to the function.
I wasn't sure where to put the prototype for cp0_raise_int5() so I
stuck it in mips64/cpu.h. If there's a better place for it, just say
so.
I also left some atomic counters for you to poke at with pstat(8) if
you want to see what the machine is doing in cp0_raise_int5(), i.e.
how often we defer clock interrupt work and how many loops you take to
arm the timer interrupt. Those will be removed before commit.
I'm running a `make build` on my EdgeRouter PoE. It only has 512MB of
RAM, so I can't do a parallel build without hanging the machine when
attempting to compile LLVM. The build has been running for four days
and the machine has not yet hung, so I think this patch is correct-ish.
I will holler if it hangs.
visa: Assuming this code looks right, could you test this on a
beefier octeon machine? Preferably a parallel build?
miod: I'm unclear whether loongson uses cp0_int5(). Am I missing
code here, or are my changes in arch/loongson sufficient?
If it's sufficient, could you test this?
I have no loongson hardware, so this is uncompiled there.
Sorry in advance if it does not compile.
Thoughts?
Index: mips64/mips64/clock.c
===================================================================
RCS file: /cvs/src/sys/arch/mips64/mips64/clock.c,v
retrieving revision 1.45
diff -u -p -r1.45 clock.c
--- mips64/mips64/clock.c 6 Apr 2022 18:59:26 -0000 1.45
+++ mips64/mips64/clock.c 31 Jul 2022 18:18:05 -0000
@@ -92,13 +92,13 @@ clockattach(struct device *parent, struc
* Interrupt handler for targets using the internal count register
* as interval clock. Normally the system is run with the clock
* interrupt always enabled. Masking is done here and if the clock
- * can not be run the tick is just counted and handled later when
- * the clock is logically unmasked again.
+ * can not be run the is tick handled later when the clock is logically
+ * unmasked again.
*/
uint32_t
cp0_int5(uint32_t mask, struct trapframe *tf)
{
- u_int32_t clkdiff;
+ u_int32_t clkdiff, pendingticks = 0;
struct cpu_info *ci = curcpu();
/*
@@ -113,15 +113,26 @@ cp0_int5(uint32_t mask, struct trapframe
}
/*
+ * If the clock interrupt is masked we can't do any work until
+ * it is unmasked.
+ */
+ if (tf->ipl >= IPL_CLOCK) {
+ ci->ci_timer_deferred = 1;
+ cp0_set_compare(cp0_get_count() - 1);
+ return CR_INT_5;
+ }
+ ci->ci_timer_deferred = 0;
+
+ /*
* Count how many ticks have passed since the last clock interrupt...
*/
clkdiff = cp0_get_count() - ci->ci_cpu_counter_last;
while (clkdiff >= ci->ci_cpu_counter_interval) {
ci->ci_cpu_counter_last += ci->ci_cpu_counter_interval;
clkdiff = cp0_get_count() - ci->ci_cpu_counter_last;
- ci->ci_pendingticks++;
+ pendingticks++;
}
- ci->ci_pendingticks++;
+ pendingticks++;
ci->ci_cpu_counter_last += ci->ci_cpu_counter_interval;
/*
@@ -132,32 +143,64 @@ cp0_int5(uint32_t mask, struct trapframe
clkdiff = cp0_get_count() - ci->ci_cpu_counter_last;
if ((int)clkdiff >= 0) {
ci->ci_cpu_counter_last += ci->ci_cpu_counter_interval;
- ci->ci_pendingticks++;
+ pendingticks++;
cp0_set_compare(ci->ci_cpu_counter_last);
}
/*
- * Process clock interrupt unless it is currently masked.
+ * Process clock interrupt.
*/
- if (tf->ipl < IPL_CLOCK) {
#ifdef MULTIPROCESSOR
- register_t sr;
+ register_t sr;
- sr = getsr();
- ENABLEIPI();
+ sr = getsr();
+ ENABLEIPI();
#endif
- while (ci->ci_pendingticks) {
- atomic_inc_long(
- (unsigned long *)&cp0_clock_count.ec_count);
- hardclock(tf);
- ci->ci_pendingticks--;
- }
+ while (pendingticks) {
+ atomic_inc_long((unsigned long *)&cp0_clock_count.ec_count);
+ hardclock(tf);
+ pendingticks--;
+ }
#ifdef MULTIPROCESSOR
- setsr(sr);
+ setsr(sr);
#endif
- }
return CR_INT_5; /* Clock is always on 5 */
+}
+
+unsigned long cp0_raise_calls, cp0_raise_miss;
+
+/*
+ * Raise the timer interrupt.
+ *
+ * We need to spin until either (a) INT5 is pending or (b) the compare
+ * register leads the count register, i.e. we know INT5 will be pending
+ * very soon.
+ *
+ * To ensure we don't spin forever, double the compensatory offset
+ * added to the compare value every time we miss the count register.
+ */
+void
+cp0_raise_int5(void)
+{
+ uint32_t compare, offset = 16;
+ int leading = 0;
+ register_t sr;
+
+ sr = disableintr();
+ while (!leading && !ISSET(cp0_get_cause(), CR_INT_5)) {
+ compare = cp0_get_count() + offset;
+ cp0_set_compare(compare);
+ leading = (int32_t)(compare - cp0_get_count()) > 0;
+ offset *= 2;
+ }
+ setsr(sr);
+
+ unsigned long misses = 0;
+ for (; offset > 32; offset /= 2)
+ misses++;
+ atomic_add_long(&cp0_raise_miss, misses);
+ atomic_inc_long(&cp0_raise_calls);
}
/*
Index: mips64/mips64/cp0access.S
===================================================================
RCS file: /cvs/src/sys/arch/mips64/mips64/cp0access.S,v
retrieving revision 1.23
diff -u -p -r1.23 cp0access.S
--- mips64/mips64/cp0access.S 1 May 2021 16:11:11 -0000 1.23
+++ mips64/mips64/cp0access.S 31 Jul 2022 18:18:05 -0000
@@ -198,3 +198,10 @@ LEAF(cpu_rnd_messybits, 0)
j ra
NOP
END(cpu_rnd_messybits)
+
+LEAF(cp0_get_cause, 0)
+ MFC0 v0, COP_0_CAUSE_REG
+ MFC0_HAZARD
+ j ra
+ NOP
+END(cp0_get_cause)
Index: mips64/include/cpu.h
===================================================================
RCS file: /cvs/src/sys/arch/mips64/include/cpu.h,v
retrieving revision 1.138
diff -u -p -r1.138 cpu.h
--- mips64/include/cpu.h 28 Jan 2022 16:20:09 -0000 1.138
+++ mips64/include/cpu.h 31 Jul 2022 18:18:05 -0000
@@ -178,11 +178,10 @@ struct cpu_info {
volatile int ci_ipl; /* software IPL */
uint32_t ci_softpending; /* pending soft interrupts */
int ci_clock_started;
+ volatile int ci_timer_deferred; /* timer interrupt postponed */
u_int32_t ci_cpu_counter_last; /* last compare value loaded */
u_int32_t ci_cpu_counter_interval; /* # of counter ticks/tick */
- u_int32_t ci_pendingticks;
-
struct pmap *ci_curpmap;
uint ci_intrdepth; /* interrupt depth */
#ifdef MULTIPROCESSOR
@@ -447,6 +446,7 @@ register_t disableintr(void);
register_t getsr(void);
register_t setsr(register_t);
+uint32_t cp0_get_cause(void);
u_int cp0_get_count(void);
register_t cp0_get_config(void);
uint32_t cp0_get_config_1(void);
@@ -461,6 +461,8 @@ void cp0_set_config(register_t);
void cp0_set_pagegrain(uint32_t);
void cp0_set_trapbase(register_t);
u_int cp1_get_prid(void);
+
+void cp0_raise_int5(void);
static inline uint32_t
cp0_get_hwrena(void)
Index: octeon/dev/octcit.c
===================================================================
RCS file: /cvs/src/sys/arch/octeon/dev/octcit.c,v
retrieving revision 1.12
diff -u -p -r1.12 octcit.c
--- octeon/dev/octcit.c 1 Sep 2019 12:16:01 -0000 1.12
+++ octeon/dev/octcit.c 31 Jul 2022 18:18:05 -0000
@@ -489,6 +489,10 @@ octcit_splx(int newipl)
(void)CIU3_RD_8(sc, CIU3_IDT_PP(CIU3_IDT(core, 0)));
}
+ /* Trigger deferred timer interrupt if clock interrupt is unmasked. */
+ if (ci->ci_timer_deferred && newipl < IPL_CLOCK)
+ cp0_raise_int5();
+
/* If we still have softints pending trigger processing. */
if (ci->ci_softpending != 0 && newipl < IPL_SOFTINT)
setsoftintr0();
Index: octeon/dev/octciu.c
===================================================================
RCS file: /cvs/src/sys/arch/octeon/dev/octciu.c,v
retrieving revision 1.17
diff -u -p -r1.17 octciu.c
--- octeon/dev/octciu.c 1 Sep 2019 12:16:01 -0000 1.17
+++ octeon/dev/octciu.c 31 Jul 2022 18:18:05 -0000
@@ -588,6 +588,10 @@ octciu_splx(int newipl)
scpu->scpu_ibank[2].en,
scpu->scpu_intem[2] & ~scpu->scpu_imask[newipl][2]);
+ /* Trigger deferred timer interrupt if clock interrupt is unmasked. */
+ if (ci->ci_timer_deferred && newipl < IPL_CLOCK)
+ cp0_raise_int5();
+
/* If we still have softints pending trigger processing. */
if (ci->ci_softpending != 0 && newipl < IPL_SOFTINT)
setsoftintr0();
Index: loongson/dev/bonito.c
===================================================================
RCS file: /cvs/src/sys/arch/loongson/dev/bonito.c,v
retrieving revision 1.35
diff -u -p -r1.35 bonito.c
--- loongson/dev/bonito.c 11 Mar 2021 11:16:57 -0000 1.35
+++ loongson/dev/bonito.c 31 Jul 2022 18:18:05 -0000
@@ -485,6 +485,11 @@ bonito_splx(int newipl)
/* Update masks to new ipl. Order highly important! */
ci->ci_ipl = newipl;
bonito_setintrmask(newipl);
+
+ /* Trigger deferred timer interrupt if clock interrupt is unmasked. */
+ if (ci->ci_timer_deferred && newipl < IPL_CLOCK)
+ cp0_raise_int5();
+
/* If we still have softints pending trigger processing. */
if (ci->ci_softpending != 0 && newipl < IPL_SOFTINT)
setsoftintr0();
Index: loongson/loongson/isa_machdep.c
===================================================================
RCS file: /cvs/src/sys/arch/loongson/loongson/isa_machdep.c,v
retrieving revision 1.4
diff -u -p -r1.4 isa_machdep.c
--- loongson/loongson/isa_machdep.c 24 Feb 2018 11:42:31 -0000 1.4
+++ loongson/loongson/isa_machdep.c 31 Jul 2022 18:18:05 -0000
@@ -88,6 +88,11 @@ loongson_isa_splx(int newipl)
/* Update masks to new ipl. Order highly important! */
ci->ci_ipl = newipl;
loongson_isa_setintrmask(newipl);
+
+ /* Trigger deferred timer interrupt if clock interrupt is unmasked. */
+ if (ci->ci_timer_deferred && newipl < IPL_CLOCK)
+ cp0_raise_int5();
+
/* If we still have softints pending trigger processing. */
if (ci->ci_softpending != 0 && newipl < IPL_SOFTINT)
setsoftintr0();
Index: loongson/loongson/loongson3_intr.c
===================================================================
RCS file: /cvs/src/sys/arch/loongson/loongson/loongson3_intr.c,v
retrieving revision 1.7
diff -u -p -r1.7 loongson3_intr.c
--- loongson/loongson/loongson3_intr.c 24 Feb 2018 11:42:31 -0000 1.7
+++ loongson/loongson/loongson3_intr.c 31 Jul 2022 18:18:05 -0000
@@ -355,6 +355,10 @@ loongson3_splx(int newipl)
REGVAL(LS3_IRT_INTENSET(0)) =
loongson3_intem & ~loongson3_imask[newipl];
+ /* Trigger deferred timer interrupt if clock interrupt is unmasked. */
+ if (ci->ci_timer_deferred && newipl < IPL_CLOCK)
+ cp0_raise_int5();
+
if (ci->ci_softpending != 0 && newipl < IPL_SOFTINT)
setsoftintr0();
}
