We have all dreamed about this for years — thank you Markus and everyone involved for finally making it happen.
On Fri, Sep 9, 2022 at 2:50 PM Markus Stange <[email protected]> wrote: > Hi all, > > I am happy to announce that crash reports just became a lot more useful! > > We now recover inlined function calls and display them in the crash stack. > In many cases, this allows you to identify the crashing code much faster > than before. In the past, many crucial stack frames would have been omitted > due to inlining. This made it borderline impossible, or at least extremely > cumbersome, to find out where the crash actually happened. > > Here's an example: > Before (inline calls lost): > https://crash-stats.mozilla.org/report/index/ffddd880-b288-49af-bfd7-1dd640220901 > After (inline calls recovered): > https://crash-stats.mozilla.org/report/index/829c31ce-e2d8-4102-9188-d35fa0220908 > In the "before" stack, you can see that the crash is inside > TenuringTracer::collectToObjectFixedPoint(), but that's where the trail > runs cold: The line number just shows you a call to traceObject, but you > don't *where inside traceObject* the crash occurred. > In the "after" stack, the stack contains two more frames inside > TenuringTracer::collectToObjectFixedPoint(): We now know about the inlined > traceObject frame, along with the line number inside that function; and > *another* frame is recovered: getDenseInitializedLength(). So the crash > really happened in getDenseInitializedLength() all along. > > This kind of multi-level inlining happens all the time, and it happens > even more so in Rust code than in C++ code. > And now you can see right through it. > > Supported versions: > The fix to emit inline information in symbol files landed last Friday in > bug 1779631 <https://bugzilla.mozilla.org/show_bug.cgi?id=1779631> and is > now riding the 106 train. This means that, as of today, inline stacks are > only available for crashes from Firefox Nightly. > > Follow-up work: > Crash *signatures* do not include inline frames yet; this work will > happen in bug 1788269 > <https://bugzilla.mozilla.org/show_bug.cgi?id=1788269>. > > Acknowledgements: > Most of this work happened in various github repos full of Rust code, > specifically in mozilla/dump_syms <https://github.com/mozilla/dump_syms/>, > getsentry/symbolic <https://github.com/getsentry/symbolic/> and > rust-minidump/rust-minidump > <https://github.com/rust-minidump/rust-minidump/>. I'd like to thank all > my reviewers for the fast reviews, especially Swatinem and loewenheim from > Sentry, and gsvelto, Gankra and willkg from Mozilla. > Furthermore, I'd like to thank Gankra and willkg for implementing the > final pieces to make the crash report infrastructure take advantage of the > new symbol information. > It's also worth noting that I didn't have to touch any C++ code when doing > this work. The Rust rewrites of dump_syms and rust-minidump have been in > production for a few months now; if I had attempted this project before > that, it would have involved a lot of C++ Google breakpad code and it would > have been much less fun. > > Background: > I worked on this project because symbol files are part of the shared > infrastructure that's used both by the crash reporter and by the profiler. > My main motivation was to get inline stacks into the profiler; we already > had them for local builds, but not for "official" builds (i.e. builds for > which symbol information comes from the symbol server). After doing all the > necessary work for the profiler, making it work for crash reports was > really straightforward and will surely pay off immensely. > > Happy crash analysis! > > -Markus > > -- > You received this message because you are subscribed to the Google Groups " > [email protected]" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/a/mozilla.org/d/msgid/dev-platform/CAL2j%2B_MyEwBey-NRSD2A_BNq8MbfAr2Ko72T-MU0DS1ycwANBw%40mail.gmail.com > <https://groups.google.com/a/mozilla.org/d/msgid/dev-platform/CAL2j%2B_MyEwBey-NRSD2A_BNq8MbfAr2Ko72T-MU0DS1ycwANBw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "[email protected]" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-platform/CADa3RMOGDoRh8Y6azrJ%2BPXria8txP4m4qMecN%3DE0wfDGOVidUQ%40mail.gmail.com.
