On 10/13/26 09:26, Gabriele Monaco wrote:
On Mon, 2026-10-12 at 18:18 +0200, Tobias Schaffner wrote:
First of all, would dropping RV in-kernel support negatively affect your
current workflow?
I am afraid it would, but first mine is a niche use case, and
second I am not very experienced with BPF, so please weigh that in.
Thanks for sharing, I'm asking this question precisely to find out all use
cases. RV is itself a niche tool, so I'd rather not squash use-cases away!
About this series, I believe we could implement the same using BPF maps
without too much effort, then the rv userspace tool could display them
nicely.
What do you think?
From an implementation perspective there may be a few pieces missing,
both in EVL and in BPF, to make this work. But I guess most of them can
use the patterns the kernel already has for NMIs, so I expect we would
find a way.
I'm curious to know how exactly you're tracing EVL right now. Are you having it
generate NMIs to linux and catch their tracepoints?
RV attaches to tracepoints and BPF would do exactly the same.
Nothing generates NMIs. But from the linux kernel's point of view any
activity of the companion core is NMI-like, since disabling interrupts
from the linux kernel does not stop it from running.
Our co-kernel keeps control of the real interrupt mask, so Linux cannot
delay it. Most of the time what Linux disables and enables is a virtual
mask instead of the real one.
Tracing already works inside EVL, thanks to a downstream patch [1]. It
is mainly fixing all the points that rely on interrupt masking by
calling the correct wrappers.
I would expect that we have to do similar for BPF. Some of the checks
already exist to cope with NMIs and we would just have to make it
recognize our stage too.
[1]
https://gitlab.com/Xenomai/linux-dovetail/-/commit/2fc39d2bb5d9ca1c0e898794fc0297eb04cf67d9
I have two more fundamental concerns though. One about verification and
one about recording dwell times.
Regarding verification:
The path I verify is deliberately kept short and simple. RV fits well
because the probe is a plain function call and the state update is a
try_cmpxchg that cannot fail, so it adds very little. A BPF program
brings more machinery into the probe path doesn't it? I fear that adding
this would force me to verify that the new RV system does not change the
behavior of the system we are trying to verify, and I suspect that could
be harder than verifying the system itself.
I had a few conversation with some Functional Safety folks over the past weeks
(the same people Daniel used to talk to), I now understand the problem better.
Their use-case was even stricter, that's why RV didn't take off for them,
essentially they're trying to prove some more isolation from interference RV
doesn't really have out of the box.
But back to the RV vs BPF case, I'm trying to reuse the exact same code in
handlers, so the verification step should be mostly equivalent.
Assuming you'd use tracepoints (BPF can have more probes but vanilla RV does
not), I guess you'd need to validate also the ftrace code that ends up calling
your handler. BPF adds a bit more on top of that too, not much though.
Then obviously there's the data representation: we use BPF maps that can be
quite complex, though simple arrays (in per-cpu/global monitors) shouldn't be.
That would probably add more to the verification you need to make..
You are clearly much more into the topic than I am. I will have to take
a closer look to better understand the BPF data structures and,
especially, their locking mechanisms before I can answer this.
My use case is, so far, not that strict. It is more that the bigger the
tool gets, the harder it becomes to argue that it does not change the
behavior of the system under test. If the tool remains simple in the end
and I can describe its behavior as deterministic, there will be no issue.
Regarding dwell timings:
The numbers you presented look very good. But wouldn't we lose
determinism by using BPF? If I want to use this as a measurement tool, I
need it either not to add latency or to add a constant amount. The mean
wake-up latency of EVL on my system is around 10 us, so an observer does
not have to be very variable before I can no longer tell its jitter from
the timings I am trying to measure.
Ok that's a good point I'd like to investigate more. BPF shows great performance
in the average case, though I fear it could have worst case spikes my quick
tests didn't show.
I'd need to run more numbers to be absolutely sure of that, I'm not aware of any
internal locks in the simple maps we are using (though I only quickly checked).
BPF maps are clearly heavier (memory indirection and/or function calls,
sometimes mitigated by the JIT). But if we properly set them up we shouldn't
expect big determinism issues.
Nevertheless, this is definitely something I need to look into.
The whole JIT thing shouldn't bring big problems either, once handlers are
compiled, they should run almost natively, but again, I am not too familiar with
everything that happens under the hood so take this with a grain of salt.
Thanks for taking a look!
It sounds to me that you think about moving the complete RV system to
BPF. Is this correct? In the end I will try to follow if you decide to
go down the BPF route. Possibly after some delay but nevertheless.
Should I wait with proposing a v3 of this patch set until you made a
final decision or do you want to merge this in the current RV system no
matter how this develops?
Best,
Tobias
Thanks,
Gabriele
