= Data and Background =
See, as some anecdotal evidence:
Bug 930797 is a user who just upgraded to Firefox 25 and is seeing these
a lot.
dmajor tracked this down to our video implementation creating many
threads, each of which has an x86 stack as well as a 1MB memory
reservation for the wow64 stack. At the point of the crash, there are
330 threads, which collectively uses about 400MB of address space just
for the thread stacks. Many of these threads appear to be created by the
Windows media platform code, although there are some Mozilla media
threads as well.
cpearce is currently looking at this.
Bug 937290 is another user who just upgraded to Firefox 25
Nobody is currently helping this user. It doesn't look consistent enough
to do easy nightly bisection, unfortunately.
There was also a bug which was showing up as an OOM crash but was in
fact a a memory corruption issue.
== Measurement ==
* Move minidump collection out of the Firefox process.
Turns out this wasn't necessary (although it would still be a good thing
to do): dmajor figured out that there was a bug in the code I originally
wrote to reserve memory for the crash reporter, and that is now fixed
and our empty-dump crash rate is very much reduced.
* Develop a tool/instructions for users to profile the VM allocations in
their Firefox process.
dmajor has some instructions for doing this using xperf at
https://etherpad.mozilla.org/sOCISHDFSm which we'll clean up and put in
a more permanent home in a bit. It does involve installing xperf and
setup and know-how, so it's not something we could give to ordinary
users. But for fairly technical users this may be a way to identify issues.
* The about:memory patches which add contiguous-vm measurement should
probably be uplifted to Fx26, and any other measurement tools that would
be valuable diagnostics.
This is done, and we also discovered that the breakpad reserved memory
was breaking this data so we fixed that in crash-stats also.
Also, we now have the ability to use DMD builds on Windows; so in the
cases where memory usage shows up in about:memory as heap-unclassified,
we can profile the memory usage that way.
Huge shout-outs to dmajor, nick nethercote, and Catalin Iacob for making
all these tools better!
== Graphics Solutions ==
The issues reported in bug 930797 at least appear to be related to HTML5
<video> rendering. The STR aren't precise, but it seems that we should
try and understand and fix the issue reported by that user. Disabling
hardware acceleration does not appear to help.
The thread issue mentioned above with video decoding appears to be at
least partly to blame for this, but we're not sure how much.
== Dealing with OOM crash sites ==
I am working on generating some reports on existing crashes where
OOMAllocationSize is variable, and also crash signatures that correlate
highly with OOM conditions. We should fix these sites.
These bugs are being filed as blockers of tracking bug 943017:
https://bugzilla.mozilla.org/showdependencytree.cgi?id=943017&hide_resolved=0
Thanks to Ehsan, Peiyong Lin, and Allessio Placitelli who have fixed
several already and are working on a few more. I'll continue to file the
most common sites as I discover them.
== New Data ==
With the empty-dump count reduced and the breakpad reservation excluded,
I have some new data from nightly builds:
https://docs.google.com/spreadsheet/ccc?key=0Ag3-54eJ7D8OdDJSV2xaeEROdjRIM1FzYmdjY1QwclE&usp=sharing
This is a dump of the Windows crash from this morning which *either*
have an OOMAllocationSize annotation indicating that jemalloc couldn't
allocate memory, *or* have less than 50MB of contiguous free VM space.
The most important columns to track are "LargestVirtualBlock",
"OOMAllocationSize", "AvailableVirtualMemory" and "AvailablePageFile".
--BDS
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
