Hi all, I am happy to announce that crash reports just became a lot more useful!
We now recover inlined function calls and display them in the crash stack. In many cases, this allows you to identify the crashing code much faster than before. In the past, many crucial stack frames would have been omitted due to inlining. This made it borderline impossible, or at least extremely cumbersome, to find out where the crash actually happened. Here's an example: Before (inline calls lost): https://crash-stats.mozilla.org/report/index/ffddd880-b288-49af-bfd7-1dd640220901 After (inline calls recovered): https://crash-stats.mozilla.org/report/index/829c31ce-e2d8-4102-9188-d35fa0220908 In the "before" stack, you can see that the crash is inside TenuringTracer::collectToObjectFixedPoint(), but that's where the trail runs cold: The line number just shows you a call to traceObject, but you don't *where inside traceObject* the crash occurred. In the "after" stack, the stack contains two more frames inside TenuringTracer::collectToObjectFixedPoint(): We now know about the inlined traceObject frame, along with the line number inside that function; and *another* frame is recovered: getDenseInitializedLength(). So the crash really happened in getDenseInitializedLength() all along. This kind of multi-level inlining happens all the time, and it happens even more so in Rust code than in C++ code. And now you can see right through it. Supported versions: The fix to emit inline information in symbol files landed last Friday in bug 1779631 <https://bugzilla.mozilla.org/show_bug.cgi?id=1779631> and is now riding the 106 train. This means that, as of today, inline stacks are only available for crashes from Firefox Nightly. Follow-up work: Crash *signatures* do not include inline frames yet; this work will happen in bug 1788269 <https://bugzilla.mozilla.org/show_bug.cgi?id=1788269>. Acknowledgements: Most of this work happened in various github repos full of Rust code, specifically in mozilla/dump_syms <https://github.com/mozilla/dump_syms/>, getsentry/symbolic <https://github.com/getsentry/symbolic/> and rust-minidump/rust-minidump <https://github.com/rust-minidump/rust-minidump/>. I'd like to thank all my reviewers for the fast reviews, especially Swatinem and loewenheim from Sentry, and gsvelto, Gankra and willkg from Mozilla. Furthermore, I'd like to thank Gankra and willkg for implementing the final pieces to make the crash report infrastructure take advantage of the new symbol information. It's also worth noting that I didn't have to touch any C++ code when doing this work. The Rust rewrites of dump_syms and rust-minidump have been in production for a few months now; if I had attempted this project before that, it would have involved a lot of C++ Google breakpad code and it would have been much less fun. Background: I worked on this project because symbol files are part of the shared infrastructure that's used both by the crash reporter and by the profiler. My main motivation was to get inline stacks into the profiler; we already had them for local builds, but not for "official" builds (i.e. builds for which symbol information comes from the symbol server). After doing all the necessary work for the profiler, making it work for crash reports was really straightforward and will surely pay off immensely. Happy crash analysis! -Markus -- You received this message because you are subscribed to the Google Groups "[email protected]" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-platform/CAL2j%2B_MyEwBey-NRSD2A_BNq8MbfAr2Ko72T-MU0DS1ycwANBw%40mail.gmail.com.
