Re: Documentation on how to read crash reports
On Wed, May 25, 2016 at 4:06 PM, Nicholas Nethercote <n.netherc...@gmail.com> wrote: > > Do we have documentation on how to read crash reports? I've finished a first pass at documentation on understanding individual crash reports. The bulk of the effort was in writing this new page: https://developer.mozilla.org/en-US/docs/Understanding_crash_reports There are numerous "XXX" comments for things I was uncertain about. If you know anything about crash reports, please read through and address any of these comments you know about, or add/fix anything else as you see fit. If you don't know anything about crash reports, please read through and let me know if anything is unclear or could otherwise be improved. I also tweaked a few other pages that talk about crash reports. Full details of what I did are here: https://bugzilla.mozilla.org/show_bug.cgi?id=1288309#c1 I have also opened https://bugzilla.mozilla.org/show_bug.cgi?id=1288310 about improving documentation relating to larger-scale analysis of clusters of crash reports. I haven't done any work on this yet. Thanks. Nick ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Documentation on how to read crash reports
On Thu, May 26, 2016, at 02:41 PM, Benoit Girard wrote: > - Crash address can give you a hint as to what's going on: 0x0 -> null > crash, 0x8 -> null offset accessing a member like this->mFoo, N.B.: due to the way addressing works on x86-64, if the crash address is "0x0" for a Linux/OS X crash report, or "0x" for a Windows crash report, it's highly likely that the value is incorrect[1]. This is due to how these faults are reported from the CPU. (I have an in-progress Breakpad patch[2] to detect this and attempt to find the actual faulting address, but it needs some more work to land upstream. The commit message also describes the root cause more thoroughly, if you're interested.) You can sanity check these crashes by clicking the "Raw Dump" tab. The top frame of the crashing stack in that JSON data will contain the registers from the crash context. If you see a suspicious value in one of them (like a poison pattern) more thorough analysis may be warranted. -Ted 1. https://bugzilla.mozilla.org/show_bug.cgi?id=974420 2. https://github.com/luser/breakpad/commit/c8c0bbcdba178b74db171c21d2c6f4a59efd0131 ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Documentation on how to read crash reports
On Thu, May 26, 2016, at 09:52 AM, Benjamin Smedberg wrote: > https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/Debugging_a_minidump I spent some time today cleaning this page up a bit. I streamlined the Windows section (since you no longer need to manually download the matching binaries or source), added a section about minidump-2-core for Linux, and added a little bit of information about some useful tools I have for additional dump analysis. Hope that's useful to someone, -Ted ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Documentation on how to read crash reports
There's some information I've learned about reading crash reports, which is obvious now but wasn't when I was an intern many years ago, that isn't really covered by these. Here's my workflow when looking at crashes: - Windows tells you if the exception occurred during a write or read. Look at the exception type. This is often the first thing I look at and important to keep in mind for the next step. - Look at the crashing source first, check the blame here and up the stack as well for a recent change. - Correlate the crash start date with the blame dates in the crashing stack and nearby. A strong match points to a regression that's easy to deal with if identified quickly. - Crash address can give you a hint as to what's going on: 0x0 -> null crash, 0x8 -> null offset accessing a member like this->mFoo, 0xfd8 -> accessing stack on some platforms, 0x80cdefd3 -> heap access, (the jemalloc poison values) - With the above look at the crash address distribution. Look at the exception type as well. If they are not all in the same bucket you *may* have a bad vtable or corruption occurring. Might want to do this per platform. - If you have a minidump see: https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/Debugging_a_minidump - If you don't have a minidump (non windows), and the line level info is insufficient, you can grab the instruction pointer and disassemble the binary. This will often tell you exactly what you were doing and which value is bad but is a bit tricky to follow with the optimizer. That's my rough workflow when looking at Soccoro. I'm leaving out the more specific searches like driver correlation for gfx crashers for instance. On Thu, May 26, 2016 at 9:52 AM, Benjamin Smedberg <benja...@smedbergs.us> wrote: > Here is a selection of docs that we've written over the years. Many are > incomplete. > > https://developer.mozilla.org/en-US/docs/Mozilla/Projects/Crash_reporting > https://developer.mozilla.org/en-US/docs/Crash_Data_Analysis > > https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/Debugging_a_minidump > > --BDS > > > On Wed, May 25, 2016 at 2:06 AM, Nicholas Nethercote < > n.netherc...@gmail.com > > wrote: > > > Hi, > > > > Do we have documentation on how to read crash reports? The only thing > > I have found is this page: > > > > https://support.mozilla.org/en-US/kb/helping-crashes > > > > which is not bad, but is lacking some details. I suspect there is > > quite a bit of crash report interpretation wisdom that is in various > > people's head, but not written down anywhere... > > > > Nick > > ___ > > dev-platform mailing list > > dev-platform@lists.mozilla.org > > https://lists.mozilla.org/listinfo/dev-platform > > > ___ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Documentation on how to read crash reports
Here is a selection of docs that we've written over the years. Many are incomplete. https://developer.mozilla.org/en-US/docs/Mozilla/Projects/Crash_reporting https://developer.mozilla.org/en-US/docs/Crash_Data_Analysis https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/Debugging_a_minidump --BDS On Wed, May 25, 2016 at 2:06 AM, Nicholas Nethercote <n.netherc...@gmail.com > wrote: > Hi, > > Do we have documentation on how to read crash reports? The only thing > I have found is this page: > > https://support.mozilla.org/en-US/kb/helping-crashes > > which is not bad, but is lacking some details. I suspect there is > quite a bit of crash report interpretation wisdom that is in various > people's head, but not written down anywhere... > > Nick > ___ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Documentation on how to read crash reports
I also found this: https://support.mozilla.org/en-US/kb/contributors-crash-course Which is similar to the first one, though presented in more of a tutorial style, and lacking screenshots. Both of them describe the meaning of individual fields, which would be better to have in crash-stats itself, so I filed bug 1275799. On Wed, May 25, 2016 at 4:06 PM, Nicholas Nethercote <n.netherc...@gmail.com> wrote: > Hi, > > Do we have documentation on how to read crash reports? The only thing > I have found is this page: > > https://support.mozilla.org/en-US/kb/helping-crashes > > which is not bad, but is lacking some details. I suspect there is > quite a bit of crash report interpretation wisdom that is in various > people's head, but not written down anywhere... > > Nick ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Documentation on how to read crash reports
Hi, Do we have documentation on how to read crash reports? The only thing I have found is this page: https://support.mozilla.org/en-US/kb/helping-crashes which is not bad, but is lacking some details. I suspect there is quite a bit of crash report interpretation wisdom that is in various people's head, but not written down anywhere... Nick ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform