Hi Jisheng,
Thanks for the clarification.
Since Radim's tracepoint code-generation results are now part of this
discussion, I've added him to Cc in case I misinterpret any aspect of
that issue.
I agree with you on one point regarding the compiler overhead. Since
the kernel supports older GCC versions, we have to take the code
generated by those supported compilers into account today. So
describing the issue simply as temporary was too strong.
However, I think there are two separate questions here: whether the
event exposes useful information, and whether the disabled-path cost
of its current implementation is acceptable.
Regarding kprobes, most of the raw fields in this event are indeed
already arguments of `__flush_tlb_range()`. My point is not that these
values are impossible to obtain dynamically.
Kprobes can be used to observe a very large part of the kernel. If the
ability to reconstruct an event with kprobes were by itself sufficient
reason not to add a tracepoint, the same reasoning would apply to many
existing tracepoints.
The distinction I see here is between observing implementation details
and exposing the semantic event directly.
For this case, the interesting information is not only `start`,
`size`, `stride`, `cmask`, etc., but their correlation with the path
Linux selected for that particular request:
local
SBI RFENCE
cross-CPU call
Calculating the local case from `cmask` may be simple with the current
implementation, but the tracing program still has to reproduce
knowledge of the selection logic. Likewise, observing downstream
functions ties the tracing program to the current call graph.
I also don't think `/proc/interrupts` provides equivalent information here.
The RISC-V IPI statistics are aggregate counters. In particular, Linux
accounts `IPI_CALL_FUNC` as "Function call interrupts", while SBI
RFENCE is handled below Linux by firmware; for example, OpenSBI
implements the remote TLB operation using its own `IPI_TLB` event.
So checking `/proc/interrupts` can certainly be useful as a sanity
check in a controlled workload: an increase in function-call IPIs can
indicate use of the Linux cross-CPU path.
But it cannot associate that IPI activity with a particular TLB
invalidation request, and it cannot correlate it with the range, ASID,
stride and target cpumask. Other users of the call-function IPI also
contribute to the same counters. In a concurrent workload this becomes
an inference rather than an observation of the TLB path decision.
That per-request correlation is what I am trying to expose with the tracepoint.
Regarding the performance argument, I agree that the disabled path
must be cheap and that the current RISC-V tracepoint code-generation
issue is relevant when evaluating the patch.
I would not claim that all tracepoints have exactly the same overhead.
My narrower point is that the issue referenced in Radim's thread is a
general interaction between RISC-V tracepoints and compiler code
generation rather than something specific to the semantics of this
event.
If the generated disabled path is considered too expensive, that is a
valid implementation concern and may require improvement or
mitigation. But I don't think it shows that the TLB path-selection
event itself is redundant because the information can be reconstructed
with kprobes.
There is also no contradiction, in my view, between using the event
for performance analysis and the event having overhead when enabled.
Tracing normally perturbs the workload when it is enabled. The
intended use here is primarily to understand things such as the
distribution of local/SBI/IPI decisions and the request context
associated with those decisions, rather than to use the tracepoint
itself for cycle-accurate measurement of the cost of a TLB flush.
The important requirement is therefore that the disabled path be
sufficiently cheap.
I think the PowerPC TLB tracepoints also remain a useful precedent
here: they show that permanent architecture-specific TLB observability
can be worthwhile when the architecture-specific information is
useful.
So I agree that convenience, maintenance cost and disabled-path
overhead all need to be considered. I just don't think that kprobes or
`/proc/interrupts` provide the same abstraction as recording the
path-selection decision directly.
Thanks,
Roman
On Wed, Oct 14, 2026 at 7:25 AM Jisheng Zhang <[email protected]> wrote:
>
> On Sun, Oct 11, 2026 at 02:43:05PM +0200, Roman Storozhenko wrote:
> > On Sat, Oct 10, 2026 at 6:44 AM Jisheng Zhang <[email protected]> wrote:
> > >
> > > On Sat, Oct 03, 2026 at 10:54:44AM +0200, Roman Storozhenko wrote:
> > > > On Fri, Oct 2, 2026 at 6:04 PM Jisheng Zhang <[email protected]> wrote:
> > > > >
> > > > > On Sun, Aug 30, 2026 at 04:36:37PM +0200, Roman 'Hedin' Storozhenko
> > > > > wrote:
> > > > > > Make RISC-V TLB flush path selection observable. Record whether
> > > > > > Linux
> > > > > > handles an invalidation locally, delegates it to SBI RFENCE, or
> > > > > > executes
> > > > > > it through a cross-CPU call, so MM activity can be correlated with
> > > > > > the
> > > > > > RISC-V, firmware, or Linux cross-CPU path carrying the request.
> > > > >
> > > > > I didn't see too much usefullness of the trace here. why not use
> > > > > kprobe
> > > > > instead? can you plz show a real usage example of the trace? Which
> > > > > bug/performance problem can be solved conveniently with this trace
> > > > > only?
> > > > >
> > > > > From another side, except x86, other architectures don't have the
> > > > > trace
> > > > > in the TLB path.
> > > > >
> > > >
> > > > Hi Jisheng,
> > > >
> > > > Thanks for the review. I agree that the current commit message describes
> > > > the intended use rather abstractly.
> > > >
> > > > A kprobe can certainly be used to investigate this. My motivation
> > > > for a tracepoint is not that dynamic tracing cannot obtain the
> > > > information, but that doing so requires knowledge of the current
> > > > implementation and reconstruction of a semantic decision made inside
> > > > __flush_tlb_range() / flush_tlb_all().
> > >
> > > I prefer aligning with other architectures(except x86), I.E not
> > > add this kind of trace. However, I would leave the decision to riscv
> > > maintainers.
> > >
> > > FYI, you may notice that the riscv trace overhead is not trivial per
> > > https://lore.kernel.org/linux-riscv/[email protected]/
> >
> > Hi Jisheng,
> >
> > Thanks for the feedback.
> >
> > Regarding the comparison with other architectures, thanks to Michael
> > Ellerman for pointing me to the PowerPC TLB tracing when we happened
> > to discuss this at LPC 2026.
> >
> > PowerPC defines the powerpc:tlbie and powerpc:tlbia trace events in
> > arch/powerpc/include/asm/trace.h, and the corresponding trace calls
> > are used in the PowerPC TLB invalidation paths.
> >
> > These events are not identical to the proposed RISC-V event: the
> > PowerPC events primarily describe architecture invalidation
> > operations, whereas this patch records the path selected by Linux
> > before dispatch. However, I think they provide useful precedent for
> > architecture-specific TLB observability beyond x86.
> >
> > Regarding kprobes, I agree that dynamic tracing can be used to investigate
> > this.
> >
> > My concern is that obtaining the same information requires knowledge
> > of the current implementation and reconstruction of the decision made
> > inside __flush_tlb_range() / flush_tlb_all().
> >
> > For example, probing __flush_tlb_range() gives us the incoming
> > request, but does not directly tell us whether Linux selected the
> > local flush, SBI RFENCE, or cross-CPU-call path. A tracing tool has to
> > probe and correlate implementation-specific downstream functions, or
> > reproduce the selection logic itself.
>
> Hmm, let's look at where the tracepoints added:
> void __flush_tlb_range(struct mm_struct *mm,
> const struct cpumask *cmask,
> unsigned long start, unsigned long size,
> unsigned long stride)
> void flush_tlb_all(void)
>
> most of the information you exported with the tracepoints are just
> parameters of flush_tlb_range.
> So kprobe or function trace can also get most of them, while whether
> local or not needs to calcuate the cmask, but getting this info is
> also easy via. adding one LoC to the kprobe routine.
>
> The tlb invalidation is frequently executed, so we need to carefully
> evaluate the convience VS maitanince offort and performance overhead when
> adding permanent tracepoints, given current riscv tracepoint overhead
> isn't trivial.
>
> BTW: whether SBI RFENCE or cross-cpu-call is much easier. Just search
> IPI in /proc/interrupts. No need to kprobe/function trace/tracepoint
>
> >
> > The proposed event records this semantic decision directly at the
> > point where Linux makes it. Users therefore do not need to depend on
> > the current helper names, call graph, or exact implementation of the
> > selection logic.
> >
> > Regarding the tracepoint overhead issue, I agree that disabled-path
> > overhead is important, especially in the TLB flush path.
> >
> > However, I see the referenced SBI tracepoint case primarily as a
> > RISC-V compiler/code-generation issue rather than as an argument
> > against providing this kind of observability.
> >
> > The referenced discussion itself shows that the generated code depends
> > significantly on compiler behavior, and that this is an area being
> > improved. The undesirable overhead is not an inherent semantic
> > requirement of the tracepoint.
> >
> > Even if the same code-generation problem currently affects this
> > tracepoint, I don't think a temporary compiler limitation should by
> > itself determine whether the tracepoint is useful or whether this
> > observability belongs in the kernel.
> >
> > If the compiler currently generates unacceptable disabled-path
> > overhead, that is certainly something that should be improved or
> > mitigated in the implementation. But I would separate that issue from
> > the question of whether exposing the RISC-V TLB path-selection
> > decision is useful.
> >
>
> But we need to live with it. Per the Radim email, gcc14 can still
> see the issue, given the minimum gcc version of kernel building
> requirement is gcc8 now, gcc14 has a long time to bailout.
>
> And another tricky point here is
> one of your motivations of adding the tracepoint is to profile then
> improve the tlb related code path performance, but adding the
> tracepoint brings non-trivial overhead to tlb code path.
--
Kind regards,
Roman 'Hedin' Storozhenko
