The entry documentation covers only the traditional irq_enter_rcu() path. It does not explain the early-counted entry contract or the retained and synthetic hardirq exit phases.
Document the staged raw count and lockdep entry, delayed traceable bookkeeping and rejected-add handling, hardirq accounting, hardirq-to-softirq classification switch, retained real IRQ exit, possible preemption and migration, and the classification-only return tail. Also cover late worker closure, conditional raw teardown and context-tracking order. Record that arm64 excludes pseudo-NMIs from this lifecycle and identify its external noinstr inlining prerequisites. Assisted-by: LLM --- Documentation/core-api/entry.rst | 89 ++++++++++++++++++++++++++++---- 1 file changed, 79 insertions(+), 10 deletions(-) diff --git a/Documentation/core-api/entry.rst b/Documentation/core-api/entry.rst index 09942135aad18..7153cc4120192 100644 --- a/Documentation/core-api/entry.rst +++ b/Documentation/core-api/entry.rst @@ -193,21 +193,14 @@ Note that the invocation of the actual interrupt handler is within a irq_enter_rcu() and irq_exit_rcu() pair. irq_enter_rcu() updates the preemption count which makes in_hardirq() -return true, handles NOHZ tick state and interrupt time accounting. This -means that up to the point where irq_enter_rcu() is invoked in_hardirq() -returns false. +return true, handles NOHZ tick state and interrupt time accounting. On this +entry path, in_hardirq() therefore returns false until irq_enter_rcu() is +invoked. irq_exit_rcu() handles interrupt time accounting, handles soft interrupts if possible, handles the NOHZ tick state and finally undoes the preemption count update. -In theory, the preemption count could be updated in irqentry_enter(). In -practice, deferring this update to irq_enter_rcu() allows the preemption-count -code to be traced, while also maintaining symmetry with irq_exit_rcu() and -irqentry_exit(), which are described in the next paragraph. The only downside -is that the early entry code up to irq_enter_rcu() must be aware that the -preemption count has not yet been updated with the HARDIRQ_OFFSET state. - Note that soft interrupt handlers must run in BH context rather than in hard interrupt context. irq_exit_rcu() therefore replaces HARDIRQ_OFFSET with SOFTIRQ_OFFSET in the preemption count while it handles soft interrupts and @@ -216,6 +209,82 @@ is still attributed to the interrupt. HARDIRQ_OFFSET is removed before irq_exit_rcu() returns because irqentry_exit() might schedule, which requires that HARDIRQ_OFFSET has been removed from the preemption count. +Pre-counted hardirq lifecycle +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Architectures which need hardirq state before early context-tracking or +tracing callbacks can use the pre-counted lifecycle. It applies only to an +ordinary hard interrupt after the architecture has distinguished it from an +NMI-like event: + + * irqentry_hardirq_enter() attempts to add HARDIRQ_OFFSET with a raw update + and records whether the update was applied. It establishes IRQ-disabled + and lockdep hardirq state before instrumentable entry work. It then runs + the generic entry and context-tracking path, including any required + transition out of an RCU extended quiescent state. + + * irq_enter_rcu_preempt_counted() later finishes the staged attempt. For an + applied update it performs traceable debug and latency bookkeeping. For a + rejected update it attempts deferred reporting and skips post-add + bookkeeping. It also performs the deferred timer repair, NOHZ tick + handling and interrupt accounting from irq_enter_rcu(). + + * irq_exit_rcu_preempt_counted_prepare() performs interrupt accounting, + softirq handling and NOHZ tick handling while carrying any applied + hardirq count across the traceable exit path. When an applied outer + hardirq invokes softirqs, both the preempt-count and lockdep + classifications temporarily switch from hardirq to softirq and back. The + architecture immediately follows the preparation with + irqentry_hardirq_exit_finish(), which performs the conditional raw-count + subtraction and lockdep teardown. + +The staged addition carries an opaque state into its instrumentable finish and +matching teardown. With DEBUG_PREEMPT and no separate reschedule bits, an +already-negative count follows the legacy DEBUG_LOCKS_WARN_ON() rule. During +an oops the warning condition is bypassed and the addition is applied. +Otherwise the attempt is rejected and leaves the count unchanged. Its finish +attempts deferred reporting, which an intervening oops can suppress. A +rejected token always skips post-add bookkeeping and the matching subtraction, +independently of whether reporting produced a warning. + +The real interrupt is complete after that teardown. Return-to-kernel +preemption can schedule and migrate the task before the architecture-specific +tail runs, so the current preemption count cannot identify the original entry +class. irqentry_state_t retains that provenance instead. + +For an ordinary hardirq returning to kernel mode, +irqentry_hardirq_tail_enter() makes a fresh staged attempt to add +HARDIRQ_OFFSET and establishes lockdep hardirq nesting after the preemption +decision. This synthetic classification covers traceable architecture tail +work, deferred timer rearming, IRQ-on preparation and context-tracking +preparation. It does not repeat interrupt accounting, RCU or tick entry. +IRQ-on preparation requires this synthetic lockdep nesting to remain present. +An interrupted outer hardirq can contribute additional nesting. + +irqentry_hardirq_tail_exit() performs the final traceable teardown preparation +and then conditionally removes any applied raw synthetic count. For an applied +staged count at outermost hardirq depth, both retained exit points recheck +regular softirq work and forced-thread timer work raised by late callbacks and +wake the corresponding workers before ending hardirq state. If the interrupted +context was in an RCU extended quiescent state, +ct_irq_exit_prepare() runs before the raw tail teardown and +ct_irq_exit_finish() completes the transition afterwards. + +On arm64 the ordinary lifecycle is selected only after the pseudo-NMI +decision. Pseudo-NMIs continue to use irqentry_nmi_enter() and +irqentry_nmi_exit(). All raw transitions are in noinstr paths, so +preempt_count(), the raw preempt-count update helpers and +__kern_my_cpu_offset() must remain inline. The arm64 conversion includes the +first requirement and depends on the `arm64 raw preempt-count inlining +prerequisite`_ and `arm64 per-CPU offset inlining prerequisite`_ for the +others. + +.. _`arm64 raw preempt-count inlining prerequisite`: + https://lore.kernel.org/all/c3a90368dd6c096af9d918e036a27728696fb8a4.1786603168.git.hongyan....@transsion.com/ + +.. _`arm64 per-CPU offset inlining prerequisite`: + https://lore.kernel.org/r/[email protected] + Even though interrupt handlers are expected to run with local interrupts disabled, interrupt nesting is common from an entry/exit perspective. For example, softirq handling happens within an irqentry_{enter,exit}() block with -- 2.39.5 (Apple Git-154)
