Re: [PATCH 0/4] tracing: improve symbolic printing
On 04/10/2023 22:43, Steven Rostedt wrote: > On Wed, 4 Oct 2023 22:35:07 +0100 > Alan Maguire wrote: > >> One thing we've heard from some embedded folks [1] is that having >> kernel BTF loadable as a separate module (rather than embedded in >> vmlinux) would help, as there are size limits on vmlinux that they can >> workaround by having modules on a different partition. We're hoping >> to get that working soon. I was wondering if you see other issues around >> BTF adoption for embedded systems that we could put on the to-do list? >> Not necessarily for this particular use-case (since there are >> complications with trace data as you describe), but just trying to make >> sure we can remove barriers to BTF adoption where possible. > > I wonder how easy is it to create subsets of BTF. For one thing, in the > future we want to be able to trace the arguments of all functions. That is, > tracing all functions at the same time (function tracer) and getting the > arguments within the trace. > > This would only require information about functions and their arguments, > which would be very useful. Is BTF easy to break apart? That is, just > generate the information needed for function arguments? > There has been a fair bit of effort around this from the userspace side; the BTF gen efforts were focused around applications carrying the minimum BTF for their needs, so just the structures needed by the particular BPF programs rather than the full set of vmlinux structures for example [1]. Parsing BTF in-kernel to pull out the BTF functions (BTF_KIND_FUNC), their prototypes (BTF_KIND_FUNC_PROTO) and all associated parameters would be pretty straightforward I think, especially if you don't need the structures that are passed via pointers. So if you're starting with the full BTF, creating a subset for use in tracing would be reasonably straightforward. My personal preference would always be to have the full BTF where possible, but if that wasn't feasible on some systems we'd need to add some options to pahole/libbpf to support such trimming during the DWARF->BTF translation process. Alan [1] https://lore.kernel.org/bpf/20220209222646.348365-7-mauri...@kinvolk.io/ > Note, pretty much all functions do not pass structures by values, and this > would not need to know the contents of a pointer to a structure. This would > mean that structure layout information is not needed. > > -- Steve >
Re: [PATCH 0/4] tracing: improve symbolic printing
On Wed, 4 Oct 2023 22:35:07 +0100 Alan Maguire wrote: > One thing we've heard from some embedded folks [1] is that having > kernel BTF loadable as a separate module (rather than embedded in > vmlinux) would help, as there are size limits on vmlinux that they can > workaround by having modules on a different partition. We're hoping > to get that working soon. I was wondering if you see other issues around > BTF adoption for embedded systems that we could put on the to-do list? > Not necessarily for this particular use-case (since there are > complications with trace data as you describe), but just trying to make > sure we can remove barriers to BTF adoption where possible. I wonder how easy is it to create subsets of BTF. For one thing, in the future we want to be able to trace the arguments of all functions. That is, tracing all functions at the same time (function tracer) and getting the arguments within the trace. This would only require information about functions and their arguments, which would be very useful. Is BTF easy to break apart? That is, just generate the information needed for function arguments? Note, pretty much all functions do not pass structures by values, and this would not need to know the contents of a pointer to a structure. This would mean that structure layout information is not needed. -- Steve
Re: [PATCH 0/4] tracing: improve symbolic printing
On 04/10/2023 18:29, Steven Rostedt wrote: > On Wed, 4 Oct 2023 09:54:31 -0700 > Jakub Kicinski wrote: > >> On Wed, 4 Oct 2023 12:35:24 -0400 Steven Rostedt wrote: Potentially naive question - the trace point holds enum skb_drop_reason. The user space can get the names from BTF. Can we not teach user space to generically look up names of enums in BTF? >>> >>> That puts a hard requirement to include BTF in builds where it was not >>> needed before. I really do not want to build with BTF just to get access to >>> these symbols. And since this is used by the embedded world, and BTF is >>> extremely bloated, the short answer is "No". >> >> Dunno. BTF is there most of the time. It could make the life of >> majority of the users far more pleasant. > > BTF isn't there for a lot of developers working in embedded who use this > code. Most my users that I deal with have minimal environments, so BTF is a > showstopper. One thing we've heard from some embedded folks [1] is that having kernel BTF loadable as a separate module (rather than embedded in vmlinux) would help, as there are size limits on vmlinux that they can workaround by having modules on a different partition. We're hoping to get that working soon. I was wondering if you see other issues around BTF adoption for embedded systems that we could put on the to-do list? Not necessarily for this particular use-case (since there are complications with trace data as you describe), but just trying to make sure we can remove barriers to BTF adoption where possible. Thanks! Alan [1] https://lore.kernel.org/bpf/CAHBbfcUkr6fTm2X9GNsFNqV75fTG=abqxfx_8ayk+4hk7he...@mail.gmail.com/ > >> >> I hope we can at least agree that the current methods of generating >> the string arrays at C level are... aesthetically displeasing. > > I don't know, I kinda like it ;-) > > -- Steve >
Re: [PATCH 0/4] tracing: improve symbolic printing
On Wed, 04 Oct 2023 20:38:46 +0200 Johannes Berg wrote: > On Wed, 2023-10-04 at 09:22 -0700, Jakub Kicinski wrote: > > > > Potentially naive question - the trace point holds enum skb_drop_reason. > > The user space can get the names from BTF. Can we not teach user space > > to generically look up names of enums in BTF? > > I'll note that, unrelated to the discussion about whether or not we > could use BTF, we couldn't do it in this case anyway since the whole > drop reasons aren't captured in enum skb_drop_reason, that contains only > the core ones, and now other subsystems are adding their own somewhat > dynamically later. Another issue with using BTF, is that the BTF would need to be saved in the trace.dat or perf.data file, as many times the trace data is moved off to another machine for offline analysis. And using the vmlinux would not be useful, because there is several times you have multiple trace files for various versions of a build and that would require mapping which vmlinux/btf file goes with which trace data. Right now, the conversions can easily be saved in the trace file directly. -- Steve
Re: [PATCH 0/4] tracing: improve symbolic printing
On Wed, 2023-10-04 at 09:22 -0700, Jakub Kicinski wrote: > > Potentially naive question - the trace point holds enum skb_drop_reason. > The user space can get the names from BTF. Can we not teach user space > to generically look up names of enums in BTF? I'll note that, unrelated to the discussion about whether or not we could use BTF, we couldn't do it in this case anyway since the whole drop reasons aren't captured in enum skb_drop_reason, that contains only the core ones, and now other subsystems are adding their own somewhat dynamically later. johannes
Re: [PATCH 0/4] tracing: improve symbolic printing
On Wed, 4 Oct 2023 09:54:31 -0700 Jakub Kicinski wrote: > On Wed, 4 Oct 2023 12:35:24 -0400 Steven Rostedt wrote: > > > Potentially naive question - the trace point holds enum skb_drop_reason. > > > The user space can get the names from BTF. Can we not teach user space > > > to generically look up names of enums in BTF? > > > > That puts a hard requirement to include BTF in builds where it was not > > needed before. I really do not want to build with BTF just to get access to > > these symbols. And since this is used by the embedded world, and BTF is > > extremely bloated, the short answer is "No". > > Dunno. BTF is there most of the time. It could make the life of > majority of the users far more pleasant. BTF isn't there for a lot of developers working in embedded who use this code. Most my users that I deal with have minimal environments, so BTF is a showstopper. > > I hope we can at least agree that the current methods of generating > the string arrays at C level are... aesthetically displeasing. I don't know, I kinda like it ;-) -- Steve
Re: [PATCH 0/4] tracing: improve symbolic printing
On Wed, 4 Oct 2023 12:35:24 -0400 Steven Rostedt wrote: > > Potentially naive question - the trace point holds enum skb_drop_reason. > > The user space can get the names from BTF. Can we not teach user space > > to generically look up names of enums in BTF? > > That puts a hard requirement to include BTF in builds where it was not > needed before. I really do not want to build with BTF just to get access to > these symbols. And since this is used by the embedded world, and BTF is > extremely bloated, the short answer is "No". Dunno. BTF is there most of the time. It could make the life of majority of the users far more pleasant. I hope we can at least agree that the current methods of generating the string arrays at C level are... aesthetically displeasing.
Re: [PATCH 0/4] tracing: improve symbolic printing
On Wed, 4 Oct 2023 09:22:05 -0700 Jakub Kicinski wrote: > Potentially naive question - the trace point holds enum skb_drop_reason. > The user space can get the names from BTF. Can we not teach user space > to generically look up names of enums in BTF? That puts a hard requirement to include BTF in builds where it was not needed before. I really do not want to build with BTF just to get access to these symbols. And since this is used by the embedded world, and BTF is extremely bloated, the short answer is "No". -- Steve
Re: [PATCH 0/4] tracing: improve symbolic printing
On Thu, 21 Sep 2023 10:51:30 +0200 Johannes Berg wrote: > So I was frustrated with not seeing the names of SKB dropreasons > for all but the core reasons, and then while looking into this > all, realized, that the current __print_symbolic() is pretty bad > anyway. > > So I came up with a new approach, using a separate declaration > of the symbols, and __print_sym() in there, but to userspace it > all doesn't matter, it shows it the same way, just dyamically > instead of munging with the strings all the time. > > This is a huge .data savings as far as I can tell, with a modest > amount (~4k) of .text addition, while making it all dynamic and > in the SKB dropreason case even reusing the existing list that > dropmonitor uses today. Surely patch 3 isn't needed here, but it > felt right. > > Anyway, I think it's a pretty reasonable approach overall, and > it does works. > > I've listed a number of open questions in the first patch since > that's where the real changes for this are. Potentially naive question - the trace point holds enum skb_drop_reason. The user space can get the names from BTF. Can we not teach user space to generically look up names of enums in BTF?
[PATCH 0/4] tracing: improve symbolic printing
So I was frustrated with not seeing the names of SKB dropreasons for all but the core reasons, and then while looking into this all, realized, that the current __print_symbolic() is pretty bad anyway. So I came up with a new approach, using a separate declaration of the symbols, and __print_sym() in there, but to userspace it all doesn't matter, it shows it the same way, just dyamically instead of munging with the strings all the time. This is a huge .data savings as far as I can tell, with a modest amount (~4k) of .text addition, while making it all dynamic and in the SKB dropreason case even reusing the existing list that dropmonitor uses today. Surely patch 3 isn't needed here, but it felt right. Anyway, I think it's a pretty reasonable approach overall, and it does works. I've listed a number of open questions in the first patch since that's where the real changes for this are. johannes
