Re: Rawhide kernel options not enabled?
Steven Rostedt writes: > [...] > But with this on, you can enable kernel function tracing at runtime. And > this is a very powerful tool. This might be something to discuss, where we > may sacrifice a bit of power for the ability of dynamic tracing. Right, but ... > Benchmarks welcome ;-) Who should shoulder that burden of proof? - FChE ___ Fedora-kernel-list mailing list Fedora-kernel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-kernel-list
Re: Rawhide kernel options not enabled?
On Wed, 4 Feb 2009, Roland McGrath wrote: > > > Or you could take my advice of many moons ago now and find less cockamamy > > > ways to implement this. ;-) > > > > By what? Rewriting gcc? > > Only a wee little bit. ;-) Seriously, -pg is eight kinds of wrong, and not > even what you really want anyway. (If your probe points are at actual > entry points instead of inside prologues, then you can get the function > arguments directly, assuming you know which calling convention that > particular function has, which you don't really but you'd probably be happy > pretending you did.) kprobes are quite expensive on the tracer to trace every function. > > You do not really need any kind of magical list of spots generated at > compile time. You can just do insertion anywhere it works. (If you're > willing to fall back to kprobes, it works most anywhere.) You can keep it > real primitive like now and just only work where there is exactly NOP5 > sitting there. Then all you're really asking for at build time is to > insert a gratuitous NOP5 at entry points. A compiler tweak for that is a > pretty simple kludge, not even tied in to actual code generation magic. > You could probably even do it with crazy-ass asm/.o fiddling as is your wont. Hacking gcc is not an option. What? Are we to wait till this hack in gcc comes out to be able to do this. Not to mention, dynamic ftrace runs on x86, powerpc, arm, and I think even superH. With more archs probably to come. Each has their own crap to deal with. The gcc for each arch will need to handle the jumps needed for modules, which on other archs is no easy task. -- Steve ___ Fedora-kernel-list mailing list Fedora-kernel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-kernel-list
Re: Rawhide kernel options not enabled?
> > Or you could take my advice of many moons ago now and find less cockamamy > > ways to implement this. ;-) > > By what? Rewriting gcc? Only a wee little bit. ;-) Seriously, -pg is eight kinds of wrong, and not even what you really want anyway. (If your probe points are at actual entry points instead of inside prologues, then you can get the function arguments directly, assuming you know which calling convention that particular function has, which you don't really but you'd probably be happy pretending you did.) You do not really need any kind of magical list of spots generated at compile time. You can just do insertion anywhere it works. (If you're willing to fall back to kprobes, it works most anywhere.) You can keep it real primitive like now and just only work where there is exactly NOP5 sitting there. Then all you're really asking for at build time is to insert a gratuitous NOP5 at entry points. A compiler tweak for that is a pretty simple kludge, not even tied in to actual code generation magic. You could probably even do it with crazy-ass asm/.o fiddling as is your wont. Thanks, Roland ___ Fedora-kernel-list mailing list Fedora-kernel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-kernel-list
Re: Rawhide kernel options not enabled?
On Wed, 4 Feb 2009, Roland McGrath wrote: > > But with this on, you can enable kernel function tracing at runtime. And > > this is a very powerful tool. This might be something to discuss, where we > > may sacrifice a bit of power for the ability of dynamic tracing. > > Or you could take my advice of many moons ago now and find less cockamamy > ways to implement this. ;-) By what? Rewriting gcc? -- Steve ___ Fedora-kernel-list mailing list Fedora-kernel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-kernel-list
Re: Rawhide kernel options not enabled?
> But with this on, you can enable kernel function tracing at runtime. And > this is a very powerful tool. This might be something to discuss, where we > may sacrifice a bit of power for the ability of dynamic tracing. Or you could take my advice of many moons ago now and find less cockamamy ways to implement this. ;-) ___ Fedora-kernel-list mailing list Fedora-kernel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-kernel-list
Re: Rawhide kernel options not enabled?
On Mon, Feb 02, 2009 at 08:20:34PM -0500, Steven Rostedt wrote: > > >On Mon, 2 Feb 2009, Steven Rostedt wrote: >> >> Do not put down that debugging ability just yet. Would it not be nice to >> just tell a user to run a script you send him (the script would do all >> the command line commands) and then the user could send you back the >> result. The user would not need to reboot or compile another kernel. You >> would be able to turn on or off anything function trace you would like. > >Also note, if your user has access to serial consoles, and many enterprise >users do, then you could also trace an oops. By setting Because many enterprise users are using Fedora... ;) >"ftrace_dump_on_oops" in the kernel command line, and have them enable >function tracing before they do whatever they do to cause the oops. The >ftrace dump output will dump to the console. If they have serial, then it >will dump to their serial console where they can record the crash. This >information can be very handy for us to analyze and find the cause on an >oops. Sounds neat. Sadly, lots of common hardware these days lacks serial ports. Will it do the same to a monitor if those options are enabled? josh ___ Fedora-kernel-list mailing list Fedora-kernel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-kernel-list
Re: Rawhide kernel options not enabled?
On Mon, 2 Feb 2009, Steven Rostedt wrote: > > Do not put down that debugging ability just yet. Would it not be nice to > just tell a user to run a script you send him (the script would do all > the command line commands) and then the user could send you back the > result. The user would not need to reboot or compile another kernel. You > would be able to turn on or off anything function trace you would like. Also note, if your user has access to serial consoles, and many enterprise users do, then you could also trace an oops. By setting "ftrace_dump_on_oops" in the kernel command line, and have them enable function tracing before they do whatever they do to cause the oops. The ftrace dump output will dump to the console. If they have serial, then it will dump to their serial console where they can record the crash. This information can be very handy for us to analyze and find the cause on an oops. -- Steve ___ Fedora-kernel-list mailing list Fedora-kernel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-kernel-list
Re: Rawhide kernel options not enabled?
On Mon, 2 Feb 2009, Josh Boyer wrote:
> On Mon, Feb 02, 2009 at 05:34:27PM -0500, Steven Rostedt wrote:
> >
> >On Mon, 2 Feb 2009, Roland McGrath wrote:
> >
> >> > I thought we were going to enable these in rawhide since the e1000
> >> > EEPROM problem was fixed:
> >> >
> >> > CONFIG_FUNCTION_TRACER (was CONFIG_FTRACE)
> >> > CONFIG_DYNAMIC_FTRACE
> >>
> >> Last I knew this still uses -pg and implies -fno-omit-frame-pointer.
> >> This probably kills performance somewhat, and more importantly during a
> >> rawhide debug-kernel phase, might change the corners of compiler behavior
> >> that we're checking vs what we'd want in a production kernel.
> >
> >The performance hit by -fno-omit-frame-pointer depends on the which
> >hardware you are running. I've been told by Arjan that the latest x86
> >hardware has negligible performance hit on this feature.
> >
> >But with this on, you can enable kernel function tracing at runtime. And
> >this is a very powerful tool. This might be something to discuss, where we
> >may sacrifice a bit of power for the ability of dynamic tracing.
>
> So I'm a kernel dude. Yet I have to wonder...
>
> What good is that to a normal user?
> Why would they ever care about being able to enable function tracing?
> Are there shiny GUI tools that do it for them?
Soon, but not yet ;-)
> Are there even release notes or wiki pages that tell them how to do it
> and why they need it?
I can write those up. Currently they just exist in
Documentation/ftrace.txt
> If it's that cool, would we ship with it enabled in stable releases?
I would expect so.
>
> I can see the value for doing kernel debugging. Telling a user "do X and
> give me the output of Y" could help kernel developers, but the normal user
> just isn't going to care.
Do not put down that debugging ability just yet. Would it not be nice to
just tell a user to run a script you send him (the script would do all
the command line commands) and then the user could send you back the
result. The user would not need to reboot or compile another kernel. You
would be able to turn on or off anything function trace you would like.
Newer features coming out in 29 that are ever cooler ;-) Like the function
graph tracer that gives you this output:
# echo sys_open > /debug/tracing/set_graph_function
# echo function_graph > /debug/tracing/current_tracer
# cat /debug/tracing/trace
# tracer: function_graph
#
# CPU DURATION FUNCTION CALLS
# | | | | | | |
1) | sys_open() {
1) |do_sys_open() {
1) | getname() {
1) 3.440 us|kmem_cache_alloc();
1) |strncpy_from_user() {
1) | __strncpy_from_user() {
1) 3.060 us|might_fault();
1) 6.845 us| }
1) + 10.508 us |}
1) + 20.774 us | }
1) | alloc_fd() {
1) |_spin_lock() {
1) 1.708 us| get_parent_ip();
1) 6.833 us|}
1) 2.032 us|expand_files();
1) |_spin_unlock() {
1) 2.024 us| get_parent_ip();
1) 7.253 us|}
1) + 24.504 us | }
1) | do_filp_open() {
1) |path_lookup_open() {
1) | get_empty_filp() {
1) 2.728 us|kmem_cache_alloc();
1) 1.834 us|get_parent_ip();
1) 1.914 us|get_parent_ip();
1) |security_file_alloc() {
1) 2.284 us| cap_file_alloc_security();
1) 6.130 us|}
1) + 24.922 us | }
1) | do_path_lookup() {
1) |_read_lock() {
1) 1.776 us| get_parent_ip();
1) 7.278 us|}
1) 2.652 us|path_get();
1) |_read_unlock() {
1) 2.034 us| get_parent_ip();
1) 6.407 us|}
1) |path_walk() {
[...]
-- Steve
___
Fedora-kernel-list mailing list
Fedora-kernel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-kernel-list
Re: Rawhide kernel options not enabled?
On Mon, Feb 02, 2009 at 05:34:27PM -0500, Steven Rostedt wrote: > >On Mon, 2 Feb 2009, Roland McGrath wrote: > >> > I thought we were going to enable these in rawhide since the e1000 >> > EEPROM problem was fixed: >> > >> > CONFIG_FUNCTION_TRACER (was CONFIG_FTRACE) >> > CONFIG_DYNAMIC_FTRACE >> >> Last I knew this still uses -pg and implies -fno-omit-frame-pointer. >> This probably kills performance somewhat, and more importantly during a >> rawhide debug-kernel phase, might change the corners of compiler behavior >> that we're checking vs what we'd want in a production kernel. > >The performance hit by -fno-omit-frame-pointer depends on the which >hardware you are running. I've been told by Arjan that the latest x86 >hardware has negligible performance hit on this feature. > >But with this on, you can enable kernel function tracing at runtime. And >this is a very powerful tool. This might be something to discuss, where we >may sacrifice a bit of power for the ability of dynamic tracing. So I'm a kernel dude. Yet I have to wonder... What good is that to a normal user? Why would they ever care about being able to enable function tracing? Are there shiny GUI tools that do it for them? Are there even release notes or wiki pages that tell them how to do it and why they need it? If it's that cool, would we ship with it enabled in stable releases? I can see the value for doing kernel debugging. Telling a user "do X and give me the output of Y" could help kernel developers, but the normal user just isn't going to care. josh ___ Fedora-kernel-list mailing list Fedora-kernel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-kernel-list
Re: Rawhide kernel options not enabled?
On Mon, 2 Feb 2009, Roland McGrath wrote: > > I thought we were going to enable these in rawhide since the e1000 > > EEPROM problem was fixed: > > > > CONFIG_FUNCTION_TRACER (was CONFIG_FTRACE) > > CONFIG_DYNAMIC_FTRACE > > Last I knew this still uses -pg and implies -fno-omit-frame-pointer. > This probably kills performance somewhat, and more importantly during a > rawhide debug-kernel phase, might change the corners of compiler behavior > that we're checking vs what we'd want in a production kernel. The performance hit by -fno-omit-frame-pointer depends on the which hardware you are running. I've been told by Arjan that the latest x86 hardware has negligible performance hit on this feature. But with this on, you can enable kernel function tracing at runtime. And this is a very powerful tool. This might be something to discuss, where we may sacrifice a bit of power for the ability of dynamic tracing. Benchmarks welcome ;-) -- Steve ___ Fedora-kernel-list mailing list Fedora-kernel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-kernel-list
Re: Rawhide kernel options not enabled?
> I thought we were going to enable these in rawhide since the e1000 > EEPROM problem was fixed: > > CONFIG_FUNCTION_TRACER (was CONFIG_FTRACE) > CONFIG_DYNAMIC_FTRACE Last I knew this still uses -pg and implies -fno-omit-frame-pointer. This probably kills performance somewhat, and more importantly during a rawhide debug-kernel phase, might change the corners of compiler behavior that we're checking vs what we'd want in a production kernel. ___ Fedora-kernel-list mailing list Fedora-kernel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-kernel-list
