Re: [PATCH] docs: reporting-issues.rst: explain how to decode stack traces
Thorsten Leemhuis writes: > Replace placeholder text about decoding stack traces with a section that > properly describes what a typical user should do these days. To make > it works for them, add a paragraph in an earlier section to ensure > people build their kernels with everything that's needed to decode stack > traces later. > > Signed-off-by: Thorsten Leemhuis > Reviewed-by: Qais Yousef > --- > v1->v2 > * Fix typo pointed out by Randy > * include review feedback from Qais and bis Reviewed-by: > > v1: > https://lore.kernel.org/lkml/20210210054823.242262-1-li...@leemhuis.info/ > --- > .../admin-guide/reporting-issues.rst | 81 ++- > 1 file changed, 59 insertions(+), 22 deletions(-) Applied, thanks. jon
Re: [PATCH] docs: reporting-issues.rst: explain how to decode stack traces
On 2/15/21 6:28 PM, Thorsten Leemhuis wrote: > Replace placeholder text about decoding stack traces with a section that > properly describes what a typical user should do these days. To make > it works for them, add a paragraph in an earlier section to ensure > people build their kernels with everything that's needed to decode stack > traces later. Looks good! > Signed-off-by: Thorsten Leemhuis > Reviewed-by: Qais Yousef Acked-by: Vlastimil Babka Thanks!
[PATCH] docs: reporting-issues.rst: explain how to decode stack traces
Replace placeholder text about decoding stack traces with a section that properly describes what a typical user should do these days. To make it works for them, add a paragraph in an earlier section to ensure people build their kernels with everything that's needed to decode stack traces later. Signed-off-by: Thorsten Leemhuis Reviewed-by: Qais Yousef --- v1->v2 * Fix typo pointed out by Randy * include review feedback from Qais and bis Reviewed-by: v1: https://lore.kernel.org/lkml/20210210054823.242262-1-li...@leemhuis.info/ --- .../admin-guide/reporting-issues.rst | 81 ++- 1 file changed, 59 insertions(+), 22 deletions(-) diff --git a/Documentation/admin-guide/reporting-issues.rst b/Documentation/admin-guide/reporting-issues.rst index 07879d01fe68..18b1280f7abf 100644 --- a/Documentation/admin-guide/reporting-issues.rst +++ b/Documentation/admin-guide/reporting-issues.rst @@ -154,8 +154,8 @@ After these preparations you'll now enter the main part: that hear about it for the first time. And if you learned something in this process, consider searching again for existing reports about the issue. - * If the failure includes a stack dump, like an Oops does, consider decoding - it to find the offending line of code. + * If your failure involves a 'panic', 'Oops', 'warning', or 'BUG', consider + decoding the kernel log to find the line of code that triggered the error. * If your problem is a regression, try to narrow down when the issue was introduced as much as possible. @@ -869,6 +869,19 @@ pick up the configuration of your current kernel and then tries to adjust it somewhat for your system. That does not make the resulting kernel any better, but quicker to compile. +Note: If you are dealing with a panic, Oops, warning, or BUG from the kernel, +please try to enable CONFIG_KALLSYMS when configuring your kernel. +Additionally, enable CONFIG_DEBUG_KERNEL and CONFIG_DEBUG_INFO, too; the +latter is the relevant one of those two, but can only be reached if you enable +the former. Be aware CONFIG_DEBUG_INFO increases the storage space required to +build a kernel by quite a bit. But that's worth it, as these options will allow +you later to pinpoint the exact line of code that triggers your issue. The +section 'Decode failure messages' below explains this in more detail. + +But keep in mind: Always keep a record of the issue encountered in case it is +hard to reproduce. Sending an undecoded report is better than not reporting +the issue at all. + Check 'taint' flag -- @@ -923,31 +936,55 @@ instead you can join. Decode failure messages --- -.. note:: +*If your failure involves a 'panic', 'Oops', 'warning', or 'BUG', consider +decoding the kernel log to find the line of code that triggered the error.* - FIXME: The text in this section is a placeholder for now and quite similar to - the old text found in 'Documentation/admin-guide/reporting-bugs.rst' - currently. It and the document it references are known to be outdated and - thus need to be revisited. Thus consider this note a request for help: if you - are familiar with this topic, please write a few lines that would fit here. - Alternatively, simply outline the current situation roughly to the main - authors of this document (see intro), as they might be able to write - something then. +When the kernel detects an internal problem, it will log some information about +the executed code. This makes it possible to pinpoint the exact line in the +source code that triggered the issue and shows how it was called. But that only +works if you enabled CONFIG_DEBUG_INFO and CONFIG_KALLSYMS when configuring +your kernel. If you did so, consider to decode the information from the +kernel's log. That will make it a lot easier to understand what lead to the +'panic', 'Oops', 'warning', or 'BUG', which increases the chances that someone +can provide a fix. - This section in the end should answer questions like "when is this actually - needed", "what .config options to ideally set earlier to make this step easy - or unnecessary?" (likely CONFIG_UNWINDER_ORC when it's available, otherwise - CONFIG_UNWINDER_FRAME_POINTER; but is there anything else needed?). +Decoding can be done with a script you find in the Linux source tree. If you +are running a kernel you compiled yourself earlier, call it like this:: -.. + [user@something ~]$ sudo dmesg | ./linux-5.10.5/scripts/decode_stacktrace.sh ./linux-5.10.5/vmlinux + +If you are running a packaged vanilla kernel, you will likely have to install +the corresponding packages with debug symbols. Then call the script (which you +might need to get from the Linux sources if your distro does not package it) +like this:: + + [user@something ~]$ sudo dmesg | ./linux-5.10.5/scripts/decode_stacktrace.sh \ +/usr/lib/debug/lib/modules/5.10.10-4.1.x86_64/vmlinux
Re: [PATCH] docs: reporting-issues.rst: explain how to decode stack traces
Hi Thorsten On 02/15/21 06:55, Thorsten Leemhuis wrote: > Hi! Many thx for looking into this, much appreciated! > > Am 14.02.21 um 17:00 schrieb Qais Yousef: > > On 02/10/21 06:48, Thorsten Leemhuis wrote: > > > >> - * If the failure includes a stack dump, like an Oops does, consider > >> decoding > >> - it to find the offending line of code. > >> + * If your failure involves a 'panic', 'oops', or 'warning', consider > >> decoding > > or 'BUG'? There are similar other places below that could benefit from this > > addition too. > > Good point. In fact there are other places in the document where this is > needed as well. Will address those in another patch. > > >> + the kernel log to find the line of code that trigger the error. > >> > >> * If your problem is a regression, try to narrow down when the issue was > >> introduced as much as possible. > >> @@ -869,6 +869,15 @@ pick up the configuration of your current kernel and > >> then tries to adjust it > >> somewhat for your system. That does not make the resulting kernel any > >> better, > >> but quicker to compile. > >> > >> +Note: If you are dealing with a kernel panic, oops, or warning, please > >> make > >> +sure to enable CONFIG_KALLSYMS when configuring your kernel. Additionally, > > > > s/make sure/try/ > > I did that, but ignored... > > > s/kernel./kernel if you can./ > > ...this. Yes, you have a point with... > > > Less demanding wording in case the user doesn't have the capability to > > rebuild > > or deploy such a kernel where the problem happens. Maybe you can tweak it > > more > > if you like too :-) > > ...that, but that section in the document is about building your own > kernel, so I'd say we don't have to be that careful here. Cool. Works for me. Thanks -- Qais Yousef
Re: [PATCH] docs: reporting-issues.rst: explain how to decode stack traces
Hi! Many thx for looking into this, much appreciated! Am 14.02.21 um 17:00 schrieb Qais Yousef: > On 02/10/21 06:48, Thorsten Leemhuis wrote: > >> - * If the failure includes a stack dump, like an Oops does, consider >> decoding >> - it to find the offending line of code. >> + * If your failure involves a 'panic', 'oops', or 'warning', consider >> decoding > or 'BUG'? There are similar other places below that could benefit from this > addition too. Good point. In fact there are other places in the document where this is needed as well. Will address those in another patch. >> + the kernel log to find the line of code that trigger the error. >> >> * If your problem is a regression, try to narrow down when the issue was >> introduced as much as possible. >> @@ -869,6 +869,15 @@ pick up the configuration of your current kernel and >> then tries to adjust it >> somewhat for your system. That does not make the resulting kernel any >> better, >> but quicker to compile. >> >> +Note: If you are dealing with a kernel panic, oops, or warning, please make >> +sure to enable CONFIG_KALLSYMS when configuring your kernel. Additionally, > > s/make sure/try/ I did that, but ignored... > s/kernel./kernel if you can./ ...this. Yes, you have a point with... > Less demanding wording in case the user doesn't have the capability to rebuild > or deploy such a kernel where the problem happens. Maybe you can tweak it more > if you like too :-) ...that, but that section in the document is about building your own kernel, so I'd say we don't have to be that careful here. >> +enable CONFIG_DEBUG_KERNEL and CONFIG_DEBUG_INFO, too; the latter is the >> +relevant one of those two, but can only be reached if you enable the >> former. Be >> +aware CONFIG_DEBUG_INFO increases the storage space required to build a >> kernel >> +by quite a bit. But that's worth it, as these options will allow you later >> to >> +pinpoint the exact line of code that triggers your issue. The section >> 'Decode >> +failure messages' below explains this in more detail. > > I think worth mentioning too that the user should keep a log of the problem > when first encountered and then attempt the above. Just in case the problem is > not reproducible easily so the info is not lost. > > Maybe something like below: > > ''' > Always keep a record of the issue encountered in case it is hard to reproduce. > Sending undecoded report is better than not sending a report at all. > ''' Very good point, added. >> +your kernel. If you did so, consider to decode the information from the >> +kernel's log. That will make it a lot easier to understand what lead to the >> +'panic', 'oops', or 'warning', which increases the chances enormously that >> +someone can provide a fix. > I suggest removing the word enormously. It helps, but it all depends on the > particular circumstances. Sometimes it does, others it doesn't. Done. > This looks good to me in general. With the above minor nits fixed, feel free > to > add my > Reviewed-by: Qais Yousef Great, thx, will do! Ciao, Thorsten
Re: [PATCH] docs: reporting-issues.rst: explain how to decode stack traces
Am 11.02.21 um 18:07 schrieb Randy Dunlap: > Just a couple of small nits (or one that is repeated): :-D > On 2/9/21 9:48 PM, Thorsten Leemhuis wrote: >> >> - * If the failure includes a stack dump, like an Oops does, consider >> decoding >> - it to find the offending line of code. >> + * If your failure involves a 'panic', 'oops', or 'warning', consider >> decoding >> + the kernel log to find the line of code that trigger the error. >triggered > […] > or it could be "code that triggers"... (just not "trigger"). Ahh, yes, you're right of course. Went with the former, many thx for taking a look and pointing it out! Ciao, Thorsten
Re: [PATCH] docs: reporting-issues.rst: explain how to decode stack traces
On 02/10/21 06:48, Thorsten Leemhuis wrote: > Replace placeholder text about decoding stack traces with a section that > properly describes what a typical user should do these days. To make > it works for them, add a paragraph in an earlier section to ensure > people build their kernels with everything that's needed to decode stack > traces later. > > Signed-off-by: Thorsten Leemhuis > --- > Reminder: This is not my area of expertise. Hopefully I didn't write anything > stupid or omitted something people find important. If I did, please let me > know, > ideally suggesting what to write; bonus points for people sending text I can > simply include in the next revision. Thanks for your effort :-) > > I CCed Sasha, because he wrote decode_stacktrace.sh; I also CCed a bunch of > people that showed interest in this topic when I asked for help on Twitter. > > I'm still unsure if linking to admin-guide/bug-hunting.rst is a good idea, as > it > seems quite outdated; reporting-bugs did that, but for now I settled on 'don't > do that'. > > Ciao, Thorsten > --- > .../admin-guide/reporting-issues.rst | 77 +-- > 1 file changed, 55 insertions(+), 22 deletions(-) > > diff --git a/Documentation/admin-guide/reporting-issues.rst > b/Documentation/admin-guide/reporting-issues.rst > index 07879d01fe68..b9c07d8e3141 100644 > --- a/Documentation/admin-guide/reporting-issues.rst > +++ b/Documentation/admin-guide/reporting-issues.rst > @@ -154,8 +154,8 @@ After these preparations you'll now enter the main part: > that hear about it for the first time. And if you learned something in > this > process, consider searching again for existing reports about the issue. > > - * If the failure includes a stack dump, like an Oops does, consider decoding > - it to find the offending line of code. > + * If your failure involves a 'panic', 'oops', or 'warning', consider > decoding or 'BUG'? There are similar other places below that could benefit from this addition too. > + the kernel log to find the line of code that trigger the error. > > * If your problem is a regression, try to narrow down when the issue was > introduced as much as possible. > @@ -869,6 +869,15 @@ pick up the configuration of your current kernel and > then tries to adjust it > somewhat for your system. That does not make the resulting kernel any better, > but quicker to compile. > > +Note: If you are dealing with a kernel panic, oops, or warning, please make > +sure to enable CONFIG_KALLSYMS when configuring your kernel. Additionally, s/make sure/try/ s/kernel./kernel if you can./ Less demanding wording in case the user doesn't have the capability to rebuild or deploy such a kernel where the problem happens. Maybe you can tweak it more if you like too :-) > +enable CONFIG_DEBUG_KERNEL and CONFIG_DEBUG_INFO, too; the latter is the > +relevant one of those two, but can only be reached if you enable the former. > Be > +aware CONFIG_DEBUG_INFO increases the storage space required to build a > kernel > +by quite a bit. But that's worth it, as these options will allow you later to > +pinpoint the exact line of code that triggers your issue. The section 'Decode > +failure messages' below explains this in more detail. > + I think worth mentioning too that the user should keep a log of the problem when first encountered and then attempt the above. Just in case the problem is not reproducible easily so the info is not lost. Maybe something like below: ''' Always keep a record of the issue encountered in case it is hard to reproduce. Sending undecoded report is better than not sending a report at all. ''' > > Check 'taint' flag > -- > @@ -923,31 +932,55 @@ instead you can join. > Decode failure messages > --- > > -.. note:: > +*If your failure involves a 'panic', 'oops', or 'warning', consider Thanks for choosing the word consider, it shouldn't be compulsory IMO. > +decoding the kernel log to find the line of code that trigger the error.* > > - FIXME: The text in this section is a placeholder for now and quite > similar to > - the old text found in 'Documentation/admin-guide/reporting-bugs.rst' > - currently. It and the document it references are known to be outdated and > - thus need to be revisited. Thus consider this note a request for help: if > you > - are familiar with this topic, please write a few lines that would fit > here. > - Alternatively, simply outline the current situation roughly to the main > - authors of this document (see intro), as they might be able to write > - something then. > +When the kernel detects an internal problem, it will log some information > about > +the executed code. This makes it possible to pinpoint the exact line in the > +source code that triggered the issue and shows how it was called. But that > only > +works if you enabled CONFIG_DEBUG_INFO and CONFIG_KALLSYMS when configuring > +your
Re: [PATCH] docs: reporting-issues.rst: explain how to decode stack traces
Hi Thorsten, Just a couple of small nits (or one that is repeated): On 2/9/21 9:48 PM, Thorsten Leemhuis wrote: > Replace placeholder text about decoding stack traces with a section that > properly describes what a typical user should do these days. To make > it works for them, add a paragraph in an earlier section to ensure > people build their kernels with everything that's needed to decode stack > traces later. > > Signed-off-by: Thorsten Leemhuis > --- > .../admin-guide/reporting-issues.rst | 77 +-- > 1 file changed, 55 insertions(+), 22 deletions(-) > > diff --git a/Documentation/admin-guide/reporting-issues.rst > b/Documentation/admin-guide/reporting-issues.rst > index 07879d01fe68..b9c07d8e3141 100644 > --- a/Documentation/admin-guide/reporting-issues.rst > +++ b/Documentation/admin-guide/reporting-issues.rst > @@ -154,8 +154,8 @@ After these preparations you'll now enter the main part: > that hear about it for the first time. And if you learned something in > this > process, consider searching again for existing reports about the issue. > > - * If the failure includes a stack dump, like an Oops does, consider decoding > - it to find the offending line of code. > + * If your failure involves a 'panic', 'oops', or 'warning', consider > decoding > + the kernel log to find the line of code that trigger the error. triggered > > * If your problem is a regression, try to narrow down when the issue was > introduced as much as possible. > @@ -869,6 +869,15 @@ pick up the configuration of your current kernel and > then tries to adjust it > somewhat for your system. That does not make the resulting kernel any better, > but quicker to compile. > > > Check 'taint' flag > -- > @@ -923,31 +932,55 @@ instead you can join. > Decode failure messages > --- > > -.. note:: > +*If your failure involves a 'panic', 'oops', or 'warning', consider > +decoding the kernel log to find the line of code that trigger the error.* triggered or it could be "code that triggers"... (just not "trigger"). -- ~Randy
[PATCH] docs: reporting-issues.rst: explain how to decode stack traces
Replace placeholder text about decoding stack traces with a section that properly describes what a typical user should do these days. To make it works for them, add a paragraph in an earlier section to ensure people build their kernels with everything that's needed to decode stack traces later. Signed-off-by: Thorsten Leemhuis --- Reminder: This is not my area of expertise. Hopefully I didn't write anything stupid or omitted something people find important. If I did, please let me know, ideally suggesting what to write; bonus points for people sending text I can simply include in the next revision. I CCed Sasha, because he wrote decode_stacktrace.sh; I also CCed a bunch of people that showed interest in this topic when I asked for help on Twitter. I'm still unsure if linking to admin-guide/bug-hunting.rst is a good idea, as it seems quite outdated; reporting-bugs did that, but for now I settled on 'don't do that'. Ciao, Thorsten --- .../admin-guide/reporting-issues.rst | 77 +-- 1 file changed, 55 insertions(+), 22 deletions(-) diff --git a/Documentation/admin-guide/reporting-issues.rst b/Documentation/admin-guide/reporting-issues.rst index 07879d01fe68..b9c07d8e3141 100644 --- a/Documentation/admin-guide/reporting-issues.rst +++ b/Documentation/admin-guide/reporting-issues.rst @@ -154,8 +154,8 @@ After these preparations you'll now enter the main part: that hear about it for the first time. And if you learned something in this process, consider searching again for existing reports about the issue. - * If the failure includes a stack dump, like an Oops does, consider decoding - it to find the offending line of code. + * If your failure involves a 'panic', 'oops', or 'warning', consider decoding + the kernel log to find the line of code that trigger the error. * If your problem is a regression, try to narrow down when the issue was introduced as much as possible. @@ -869,6 +869,15 @@ pick up the configuration of your current kernel and then tries to adjust it somewhat for your system. That does not make the resulting kernel any better, but quicker to compile. +Note: If you are dealing with a kernel panic, oops, or warning, please make +sure to enable CONFIG_KALLSYMS when configuring your kernel. Additionally, +enable CONFIG_DEBUG_KERNEL and CONFIG_DEBUG_INFO, too; the latter is the +relevant one of those two, but can only be reached if you enable the former. Be +aware CONFIG_DEBUG_INFO increases the storage space required to build a kernel +by quite a bit. But that's worth it, as these options will allow you later to +pinpoint the exact line of code that triggers your issue. The section 'Decode +failure messages' below explains this in more detail. + Check 'taint' flag -- @@ -923,31 +932,55 @@ instead you can join. Decode failure messages --- -.. note:: +*If your failure involves a 'panic', 'oops', or 'warning', consider +decoding the kernel log to find the line of code that trigger the error.* - FIXME: The text in this section is a placeholder for now and quite similar to - the old text found in 'Documentation/admin-guide/reporting-bugs.rst' - currently. It and the document it references are known to be outdated and - thus need to be revisited. Thus consider this note a request for help: if you - are familiar with this topic, please write a few lines that would fit here. - Alternatively, simply outline the current situation roughly to the main - authors of this document (see intro), as they might be able to write - something then. +When the kernel detects an internal problem, it will log some information about +the executed code. This makes it possible to pinpoint the exact line in the +source code that triggered the issue and shows how it was called. But that only +works if you enabled CONFIG_DEBUG_INFO and CONFIG_KALLSYMS when configuring +your kernel. If you did so, consider to decode the information from the +kernel's log. That will make it a lot easier to understand what lead to the +'panic', 'oops', or 'warning', which increases the chances enormously that +someone can provide a fix. - This section in the end should answer questions like "when is this actually - needed", "what .config options to ideally set earlier to make this step easy - or unnecessary?" (likely CONFIG_UNWINDER_ORC when it's available, otherwise - CONFIG_UNWINDER_FRAME_POINTER; but is there anything else needed?). +Decoding can be done with a script you find in the Linux source tree. If you +are running a kernel you compiled yourself earlier, call it like this:: -.. + [user@something ~]$ sudo dmesg | ./linux-5.10.5/scripts/decode_stacktrace.sh ./linux-5.10.5/vmlinux + +If you are running a packaged vanilla kernel, you will likely have to install +the corresponding packages with debug symbols. Then call the script (which you +might need to get from the Linux sources if your distro