Ordinary EL0 and EL1 interrupt paths establish generic entry state before irq_enter_rcu() adds HARDIRQ_OFFSET and lockdep nesting. Callbacks in that early window are therefore not classified as hardirq work. Their exit paths also drop the classification before function-graph return callbacks and the kernel return tail have completed.
After the EL1 pseudo-NMI decision, enter ordinary interrupts through irqentry_hardirq_enter(). Finish the staged count update and delayed timer and accounting work with irq_enter_rcu_preempt_counted(). Use the retained exit preparation followed immediately by conditional raw count and lockdep teardown. For kernel returns, preserve the existing per-CPU register adjustment, then stage hardirq return-tail classification before MTE exit tracing. Keep the pseudo-NMI lifecycle unchanged. The noinstr path also depends on the raw preempt-count and __kern_my_cpu_offset() inlining prerequisites earlier in this series. Link: https://lore.kernel.org/all/c3a90368dd6c096af9d918e036a27728696fb8a4.1786603168.git.hongyan....@transsion.com/ Link: https://lore.kernel.org/r/[email protected] Assisted-by: LLM --- arch/arm64/kernel/entry-common.c | 28 ++++++++++++++++++++++------ 1 file changed, 22 insertions(+), 6 deletions(-) diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index 003f84109a913..d70cffffe71c6 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -94,6 +94,8 @@ static void noinstr __arm64_exit_to_kernel_mode(struct pt_regs *regs, { local_daif_mask(); irqentry_exit_pcpu_adjust(regs); + /* arm64 uses the split exit API, so enter the tail before MTE tracing. */ + irqentry_hardirq_tail_enter(&state); mte_check_tfsr_exit(); irqentry_exit_to_kernel_mode_after_preempt(regs, state); } @@ -560,13 +562,19 @@ static __always_inline void __el1_pnmi(struct pt_regs *regs, static __always_inline void __el1_irq(struct pt_regs *regs, void (*handler)(struct pt_regs *)) { + struct irqentry_hardirq_cookie cookie; irqentry_state_t state; + bool do_sub; - state = arm64_enter_from_kernel_mode(regs); + state = irqentry_hardirq_enter(regs); + cookie = irqentry_hardirq_count_state(state); + mte_check_tfsr_entry(); + mte_disable_tco_entry(current); - irq_enter_rcu(); + irq_enter_rcu_preempt_counted(cookie); do_interrupt_handler(regs, handler); - irq_exit_rcu(); + do_sub = irq_exit_rcu_preempt_counted_prepare(cookie); + irqentry_hardirq_exit_finish(do_sub); arm64_exit_to_kernel_mode_preempt(regs, state); } @@ -877,16 +885,24 @@ asmlinkage void noinstr el0t_64_sync_handler(struct pt_regs *regs) static void noinstr el0_interrupt(struct pt_regs *regs, void (*handler)(struct pt_regs *)) { - arm64_enter_from_user_mode(regs); + struct irqentry_hardirq_cookie cookie; + irqentry_state_t state; + bool do_sub; + + state = irqentry_hardirq_enter(regs); + cookie = irqentry_hardirq_count_state(state); + mte_disable_tco_entry(current); + sme_enter_from_user_mode(); write_sysreg(DAIF_PROCCTX_NOIRQ, daif); if (regs->pc & BIT(55)) arm64_apply_bp_hardening(); - irq_enter_rcu(); + irq_enter_rcu_preempt_counted(cookie); do_interrupt_handler(regs, handler); - irq_exit_rcu(); + do_sub = irq_exit_rcu_preempt_counted_prepare(cookie); + irqentry_hardirq_exit_finish(do_sub); arm64_exit_to_user_mode(regs); } -- 2.39.5 (Apple Git-154)
