> On Nov 10, 2025, at 11:59 AM, Steven Rostedt <[email protected]> wrote: > > On Mon, 10 Nov 2025 10:16:45 -0500 > Joel Fernandes <[email protected]> wrote: > >>> IMO, the best way to do this is the tracing subsystem. It is the lowest >>> overhead >>> runtime kernel logging system that I know off, lockless, independent of the >>> serial console etc, next to no runtime overhead when off, etc. >>> >>> I recommend we use the tracing subsystem for "trace" and even "spam" level >>> logging levels for Nova. The brave souls can always ask the tracing >>> subsystem to >>> also spam to kernel logs if they so wish. >>> >>> ++ Tracing Czar Steven Rostedt as well. Steve, Nova is a new modern Nvidia >>> GPU >>> driver. > > Not sure if there was a question here, but you can enable tracing via the > kernel command line and it will record into the ring buffer, and read it > out via /sys/kernel/tracing/trace. You could also create instances by the > kernel command line that will place events in different ring buffer > instances (/sys/kernel/tracing/instances/<instance>/trace). You could even > filter the events based on the trace event fields, process ID, CPU, etc. > > There's also a tp_printk kernel command line option that will turn every > trace event into a printk() statement. You need to be careful of what > events you enable when doing this, as some events (like locking events) can > live lock the system if they are piped to printk(). > > After boot up, you can turn off the tp_printk with: > > echo 0 > /proc/sys/kernel/tracepoint_printk
Thanks Steve. The reason I also added you was also to keep me honest about tracing performance for logging that is noisy. The context is dev_dbg noise can have strong side effects that tracing will for sure circumvent. I would keep dmesg printing only for info/error/warning with tracing for the noisy ones. Maybe we should add a facility to dynamically turn all dev_dbg into tracing… Thanks, - Joel > > -- Steve > >>> >>> I guess we have to decide how to do this - what kind of tracepoints do we >>> need >>> for Nova. One use case that just came up is RPC message buffer dumps for >>> debugging communication with the firmware.
