Re: Website memory leaks

[Nick Fitzgerald]

On Mon, Nov 6, 2017 at 1:53 AM, David Teller  wrote:

> I wanted to add something like that to about:performance, but at the
> time, my impression was that we did not have sufficient platform data on
> where allocations come from to provide something convincing.
>

​The SpiderMonkey Debugger API has hooks to capture the JS stack with some
probability for each JS object allocation:
https://developer.mozilla.org/en-US/docs/Tools/Debugger-API/Debugger.Memory#Allocation_Site_Tracking

The probability allows you to dial up or down the overhead vs amount of
data trade off.

The API was designed in such a way that it doesn't disable ion compilation
when enabled. It's not somethign we' want on all the time for everyone, but
should have low enough overhead that we could turn it on in something like
about:performance.

Would be great to expand that API to observe JS string allocations as
well.​​ C++ allocations could be useful too, but there are a lot more open
questions there.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Website memory leaks

[David Teller]

As a user, I would definitely love to have this.

I wanted to add something like that to about:performance, but at the
time, my impression was that we did not have sufficient platform data on
where allocations come from to provide something convincing.

Cheers,
 David

On 02/11/17 15:34, Randell Jesup wrote:
> [Note: I'm a tab-hoarder - but that doesn't really cause this problem]
> 
> tl;dr: we should look at something (roughly) like the existing "page is
> making your browser slow" dialog for website leaks.
> 
> 
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Website memory leaks

[Nick Fitzgerald]

We have https://bugzilla.mozilla.org/show_bug.cgi?id=1243091 on file for
automatic leak detection in the DevTools' memory panel.

I'd have liked to try implementing
http://www.cs.utexas.edu/ftp/techreports/tr06-07.pdf because it can see
through frameworks/libraries (or claims to in a convincing way, at least)
and point to the "user" code that is transitively responsible. In today's
modern web that is full of every kind of framework and library, that seems
like a very useful property to have.

As far as seeing the CC's heap, there is
https://bugzilla.mozilla.org/show_bug.cgi?id=1057057 on file for that. It
shouldn't be hard for someone who understand the CC and all of its macros.
You basically just make a specialization of JS::ubi::Concrete that can
enumerate outgoing edges and report shallow size.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Website memory leaks

[Robert O'Callahan]

Now that I'm writing a Web app for real, I realize just how easy it is to
accidentally leak :-(.

It would be useful, or at least cool, to be able to show users and
developers a graph of memory usage over time, one line per tab. You could
limit this to just show the top N memory-hungry tabs.

A UI intervention like the slow-script notification seems plausible. You
can probably come up with pretty good heuristics for identifying leaking
tabs. One signal would be steady memory growth during times without user
interaction. Then you can show the memory growth graph and offer to reload
the tab.

When devtools are open you could be a lot more aggressive about warning
developers if you think their app might be leaking. Actually, as an aside,
there's a more general devtools request here: when I'm using my own app,
even when devtools are not open, I'd like to be notified of JS errors and
other serious issues. One way to do this would be that if I ever open
devtools on a site, then set a "warn on errors" flag for that site, and
ensure the warnings offer a "stop warning me" button.

There's a lot of research literature on tools for tracking down leaks using
periodic heap snapshots and other techniques. Most of it's probably junk
but it's worth consulting. Nick Mitchell and Gary Sevitsky did some good
work.

IIRC you already have a "this is probably an ad" URL blacklist, so bounding
the memory usage of those IFRAMEs, freezing them when they reach it, sounds
good. You shouldn't need process isolation for IFRAMEs for this.

Rob
-- 
lbir ye,ea yer.tnietoehr  rdn rdsme,anea lurpr  edna e hnysnenh hhe uresyf
toD
selthor  stor  edna  siewaoeodm  or v sstvr  esBa  kbvted,t
rdsme,aoreseoouoto
o l euetiuruewFa  kbn e hnystoivateweh uresyf tulsa rehr  rdm  or rnea
lurpr
.a war hsrer holsa rodvted,t  nenh hneireseoouot.tniesiewaoeivatewt sstvr
esn
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Website memory leaks

[smaug]

On 11/02/2017 09:58 PM, Kris Maglione wrote:

Related: I've been thinking for a long time that we need better tools for 
tracking what sites/usage patterns are responsible for the time we spend in
CC (and possibly GC, but I think that tends to be less of a problem).

I've noticed, in particular, that having multiple Amazon tabs open, and keeping 
them open for a long time, tends to lead to many, long CC pauses. But
I have no idea why, or how to even begin profiling it. And the only reason I 
even know/suspect that Amazon is the problem is that I've noticed a
strong pattern.


Have you filed a bug? Web sites do often leak, and by leak I mean keeping tons 
of such objects alive that they don't need anymore. If CC times go up
when a web site is leaking, it probably means a missing "black-bit-propagation" 
optimization.
Google Reader was a canonical example of a web page which leaked basically 
everything, and we did optimize things out from CC graph so that
at least CC times stayed low enough, even though memory usage went up.



Leaks certainly make the problem worse, but I'm not sure that they're all of the 
problem. Either way, I think being able to say "this compartment n
milliseconds of CC time in the last minute" would be a pretty good way for us 
to pick out memory problems in particular web pages.


On Thu, Nov 02, 2017 at 10:34:22AM -0400, Randell Jesup wrote:

[Note: I'm a tab-hoarder - but that doesn't really cause this problem]

tl;dr: we should look at something (roughly) like the existing "page is
making your browser slow" dialog for website leaks.


Over the last several months (and maybe the last year), I've noticed an
increasing problem in websites: runaway leaks, leading to
a poorly-performing browser or OOM crashes.  Yes, these have *always*
occurred (people write bad code, shock!), but the incidence seems to
have gotten much worse 'recently' (anecdotally).

This may be the result of ads/ad networks (in some cases it provably
is), in other cases it's the sites themselves (like the 4M copies of
":DIV" that facebook was creating over a day if you left a message
visible, plus GB of related leaks (80K copies of many strings/objects).
Many of the pages causing these leaks are major sites, like nytimes.com,
washington post, cnn, arstechnica, Atlantic, New Yorker, etc.

However, regardless of who's making the error or the details, I've been
noticing that I'm having to "hunt for the bad tab" more and more
frequently (usually via about:memory, which users wouldn't
do/understand).  Multi-e10s helps a bit and some tabs won't degrade, but
also enables my system to get into phyical-memory-pressure faster.  (The
machine I notice this on is Win10, 12GB ram, quad-core AMD 8xxx (older,
but 3.5GHz).  I'm running Nightly (32-bit) with one profile (for
facebook), and beta (64bit) with another profile).

While physical-memory pressure causes problems (including for the system
as a whole), the leaking can make Content processes unresponsive, to the
point where about:memory doesn't get data for them (or it's incomplete).
This makes it hard-to-impossible to fix the problem; I kill Content
processes by hand in that case - regular users would just force-close the
browser (and perhaps start Chrome instead of restarting.)  We see an
insanely high number of OOMs in the field; likely this is a major
contributing factor.  Chrome will *tend* to have less tabs impacted by
one leaking, though the leaking tab will still be ugly.

Sometimes the leak manifests simply as a leak (like the washington post
tab I just killed and got 3.5 of the 5GB in use by a content process
back).  Others (depending I assume on what is leaked and how it's kept
alive) cause a core (each) to be chewed doing continual GC/CC (or
processing events in app JS touching some insanely-long internal
structure), slowing the process (and system) to a crawl.


*Generally* these are caused by leaving a page up for a while - navigate
and leave, and you never notice it (part of why website developers don't
fix these, or perhaps don't care (much)).  But even walking away from a
machine with one or a couple of tabs, and coming back the next day can
present you with an unusable tab/browser, or a "this tab has crashed"
screen.


Hoping site owners (or worse, ad producers) will fix their leaks is a
losing game, though we should encourage it and offer tools where
possible.  However, we need a broader solution (or at least tools) for
dealing with this for users.


I don't have a magic-bullet solution, and I'm sure there isn't one given
JS and GC as a core of browsers.  I do think we need to talk about it
(and perhaps not just Mozilla), and find some way to help users here.


One half-baked idea is to provide a UI intervention similar to what we
do on JS that runs too-long, and ask the user if they want to freeze the
tab (or better yet in this case, reload it).  If we trigger before the
side-effects get out of hand, freezing may be ok.  We should also
identify the 

Re: Website memory leaks

[Randell Jesup]

>On Thu, Nov 02, 2017 at 05:37:30PM +0200, smaug wrote:
>>This has been an issue forever, and there aren't really good tools on any 
>>browser, as far as
>>I know, for web devs to debug their leaks.
>>Internally we do have useful data (CC and GC graphs and such), but would need 
>>quite some ux skills to design some good
>>UI to deal with leaks. Also, the data to deal with is often massive, so the 
>>tool should be implemented that in mind.
>
>We do have memory tools in devtools now, and the dominator tree in
>particular can provide some useful data for tracking down leaks. But those
>tools aren't especially well-maintained at the moment, and I honestly think
>they're too flaky at this point to recommend to webdevs for general use :(
>
>It would be nice if we could prioritize them again.

Also, tools for developing a page are one thing, but usage in the field
is another.  Ability to know in a loaded page about memory use (and to
know about usage of an embedded iframe) would give web devs information
and a way to put their own pressure (and maybe limits) on.

Plus, most devs unless they're dealing with a report about a problem
won't go looking.  Something that proactively can poke them is far more
likely to get action.

-- 
Randell Jesup, Mozilla Corp
remove "news" for personal email
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Website memory leaks

[smaug]

On 11/02/2017 10:01 PM, Kris Maglione wrote:

On Thu, Nov 02, 2017 at 05:37:30PM +0200, smaug wrote:

This has been an issue forever, and there aren't really good tools on any 
browser, as far as
I know, for web devs to debug their leaks.
Internally we do have useful data (CC and GC graphs and such), but would need 
quite some ux skills to design some good
UI to deal with leaks. Also, the data to deal with is often massive, so the 
tool should be implemented that in mind.


We do have memory tools in devtools now, and the dominator tree

We've had that for quite some time, but that is missing the native side of the 
object graph, which is often rather crucial.
And I think it is hard to query various things from the tool.



in particular can provide some useful data for tracking down leaks. But those 
tools
aren't especially well-maintained at the moment, and I honestly think they're 
too flaky at this point to recommend to webdevs for general use :(

It would be nice if we could prioritize them again.


On 11/02/2017 04:34 PM, Randell Jesup wrote:

[Note: I'm a tab-hoarder - but that doesn't really cause this problem]

tl;dr: we should look at something (roughly) like the existing "page is
making your browser slow" dialog for website leaks.


Over the last several months (and maybe the last year), I've noticed an
increasing problem in websites: runaway leaks, leading to
a poorly-performing browser or OOM crashes.  Yes, these have *always*
occurred (people write bad code, shock!), but the incidence seems to
have gotten much worse 'recently' (anecdotally).

This may be the result of ads/ad networks (in some cases it provably
is), in other cases it's the sites themselves (like the 4M copies of
":DIV" that facebook was creating over a day if you left a message
visible, plus GB of related leaks (80K copies of many strings/objects).
Many of the pages causing these leaks are major sites, like nytimes.com,
washington post, cnn, arstechnica, Atlantic, New Yorker, etc.

However, regardless of who's making the error or the details, I've been
noticing that I'm having to "hunt for the bad tab" more and more
frequently (usually via about:memory, which users wouldn't
do/understand).  Multi-e10s helps a bit and some tabs won't degrade, but
also enables my system to get into phyical-memory-pressure faster.  (The
machine I notice this on is Win10, 12GB ram, quad-core AMD 8xxx (older,
but 3.5GHz).  I'm running Nightly (32-bit) with one profile (for
facebook), and beta (64bit) with another profile).

While physical-memory pressure causes problems (including for the system
as a whole), the leaking can make Content processes unresponsive, to the
point where about:memory doesn't get data for them (or it's incomplete).
This makes it hard-to-impossible to fix the problem; I kill Content
processes by hand in that case - regular users would just force-close the
browser (and perhaps start Chrome instead of restarting.)  We see an
insanely high number of OOMs in the field; likely this is a major
contributing factor.  Chrome will *tend* to have less tabs impacted by
one leaking, though the leaking tab will still be ugly.

Sometimes the leak manifests simply as a leak (like the washington post
tab I just killed and got 3.5 of the 5GB in use by a content process
back).  Others (depending I assume on what is leaked and how it's kept
alive) cause a core (each) to be chewed doing continual GC/CC (or
processing events in app JS touching some insanely-long internal
structure), slowing the process (and system) to a crawl.


*Generally* these are caused by leaving a page up for a while - navigate
and leave, and you never notice it (part of why website developers don't
fix these, or perhaps don't care (much)).  But even walking away from a
machine with one or a couple of tabs, and coming back the next day can
present you with an unusable tab/browser, or a "this tab has crashed"
screen.


Hoping site owners (or worse, ad producers) will fix their leaks is a
losing game, though we should encourage it and offer tools where
possible.  However, we need a broader solution (or at least tools) for
dealing with this for users.


I don't have a magic-bullet solution, and I'm sure there isn't one given
JS and GC as a core of browsers.  I do think we need to talk about it
(and perhaps not just Mozilla), and find some way to help users here.


One half-baked idea is to provide a UI intervention similar to what we
do on JS that runs too-long, and ask the user if they want to freeze the
tab (or better yet in this case, reload it).  If we trigger before the
side-effects get out of hand, freezing may be ok.  We should also
identify the tab (in the message/popup, and visually in the tab bar - we
don't do either today).  This solution has issues (though freezing tabs
now that "slow down your browser" does too, but it's still useful
overall).

We can also look at tools to characterize site leaks or identify
patterns that point to a leak before it gets out of 

Re: Website memory leaks

[Kris Maglione]

On Thu, Nov 02, 2017 at 05:37:30PM +0200, smaug wrote:

This has been an issue forever, and there aren't really good tools on any 
browser, as far as
I know, for web devs to debug their leaks.
Internally we do have useful data (CC and GC graphs and such), but would need 
quite some ux skills to design some good
UI to deal with leaks. Also, the data to deal with is often massive, so the 
tool should be implemented that in mind.


We do have memory tools in devtools now, and the dominator tree in 
particular can provide some useful data for tracking down leaks. But 
those tools aren't especially well-maintained at the moment, and I 
honestly think they're too flaky at this point to recommend to webdevs 
for general use :(


It would be nice if we could prioritize them again.


On 11/02/2017 04:34 PM, Randell Jesup wrote:

[Note: I'm a tab-hoarder - but that doesn't really cause this problem]

tl;dr: we should look at something (roughly) like the existing "page is
making your browser slow" dialog for website leaks.


Over the last several months (and maybe the last year), I've noticed an
increasing problem in websites: runaway leaks, leading to
a poorly-performing browser or OOM crashes.  Yes, these have *always*
occurred (people write bad code, shock!), but the incidence seems to
have gotten much worse 'recently' (anecdotally).

This may be the result of ads/ad networks (in some cases it provably
is), in other cases it's the sites themselves (like the 4M copies of
":DIV" that facebook was creating over a day if you left a message
visible, plus GB of related leaks (80K copies of many strings/objects).
Many of the pages causing these leaks are major sites, like nytimes.com,
washington post, cnn, arstechnica, Atlantic, New Yorker, etc.

However, regardless of who's making the error or the details, I've been
noticing that I'm having to "hunt for the bad tab" more and more
frequently (usually via about:memory, which users wouldn't
do/understand).  Multi-e10s helps a bit and some tabs won't degrade, but
also enables my system to get into phyical-memory-pressure faster.  (The
machine I notice this on is Win10, 12GB ram, quad-core AMD 8xxx (older,
but 3.5GHz).  I'm running Nightly (32-bit) with one profile (for
facebook), and beta (64bit) with another profile).

While physical-memory pressure causes problems (including for the system
as a whole), the leaking can make Content processes unresponsive, to the
point where about:memory doesn't get data for them (or it's incomplete).
This makes it hard-to-impossible to fix the problem; I kill Content
processes by hand in that case - regular users would just force-close the
browser (and perhaps start Chrome instead of restarting.)  We see an
insanely high number of OOMs in the field; likely this is a major
contributing factor.  Chrome will *tend* to have less tabs impacted by
one leaking, though the leaking tab will still be ugly.

Sometimes the leak manifests simply as a leak (like the washington post
tab I just killed and got 3.5 of the 5GB in use by a content process
back).  Others (depending I assume on what is leaked and how it's kept
alive) cause a core (each) to be chewed doing continual GC/CC (or
processing events in app JS touching some insanely-long internal
structure), slowing the process (and system) to a crawl.


*Generally* these are caused by leaving a page up for a while - navigate
and leave, and you never notice it (part of why website developers don't
fix these, or perhaps don't care (much)).  But even walking away from a
machine with one or a couple of tabs, and coming back the next day can
present you with an unusable tab/browser, or a "this tab has crashed"
screen.


Hoping site owners (or worse, ad producers) will fix their leaks is a
losing game, though we should encourage it and offer tools where
possible.  However, we need a broader solution (or at least tools) for
dealing with this for users.


I don't have a magic-bullet solution, and I'm sure there isn't one given
JS and GC as a core of browsers.  I do think we need to talk about it
(and perhaps not just Mozilla), and find some way to help users here.


One half-baked idea is to provide a UI intervention similar to what we
do on JS that runs too-long, and ask the user if they want to freeze the
tab (or better yet in this case, reload it).  If we trigger before the
side-effects get out of hand, freezing may be ok.  We should also
identify the tab (in the message/popup, and visually in the tab bar - we
don't do either today).  This solution has issues (though freezing tabs
now that "slow down your browser" does too, but it's still useful
overall).

We can also look at tools to characterize site leaks or identify
patterns that point to a leak before it gets out of hand.  Also tools
that help website builders identify if their pages are leaking in the
field (memory-use triggers? within-tab about:memory dumps available to
the page?)

Perhaps we can also push to limit memory use (CPU use??) in 

Re: Website memory leaks

[Kris Maglione]

Related: I've been thinking for a long time that we need better tools 
for tracking what sites/usage patterns are responsible for the time we 
spend in CC (and possibly GC, but I think that tends to be less of a 
problem).


I've noticed, in particular, that having multiple Amazon tabs open, and 
keeping them open for a long time, tends to lead to many, long CC 
pauses. But I have no idea why, or how to even begin profiling it. And 
the only reason I even know/suspect that Amazon is the problem is that 
I've noticed a strong pattern.


Leaks certainly make the problem worse, but I'm not sure that they're 
all of the problem. Either way, I think being able to say "this 
compartment n milliseconds of CC time in the last minute" would be a 
pretty good way for us to pick out memory problems in particular web 
pages.



On Thu, Nov 02, 2017 at 10:34:22AM -0400, Randell Jesup wrote:

[Note: I'm a tab-hoarder - but that doesn't really cause this problem]

tl;dr: we should look at something (roughly) like the existing "page is
making your browser slow" dialog for website leaks.


Over the last several months (and maybe the last year), I've noticed an
increasing problem in websites: runaway leaks, leading to
a poorly-performing browser or OOM crashes.  Yes, these have *always*
occurred (people write bad code, shock!), but the incidence seems to
have gotten much worse 'recently' (anecdotally).

This may be the result of ads/ad networks (in some cases it provably
is), in other cases it's the sites themselves (like the 4M copies of
":DIV" that facebook was creating over a day if you left a message
visible, plus GB of related leaks (80K copies of many strings/objects).
Many of the pages causing these leaks are major sites, like nytimes.com,
washington post, cnn, arstechnica, Atlantic, New Yorker, etc.

However, regardless of who's making the error or the details, I've been
noticing that I'm having to "hunt for the bad tab" more and more
frequently (usually via about:memory, which users wouldn't
do/understand).  Multi-e10s helps a bit and some tabs won't degrade, but
also enables my system to get into phyical-memory-pressure faster.  (The
machine I notice this on is Win10, 12GB ram, quad-core AMD 8xxx (older,
but 3.5GHz).  I'm running Nightly (32-bit) with one profile (for
facebook), and beta (64bit) with another profile).

While physical-memory pressure causes problems (including for the system
as a whole), the leaking can make Content processes unresponsive, to the
point where about:memory doesn't get data for them (or it's incomplete).
This makes it hard-to-impossible to fix the problem; I kill Content
processes by hand in that case - regular users would just force-close the
browser (and perhaps start Chrome instead of restarting.)  We see an
insanely high number of OOMs in the field; likely this is a major
contributing factor.  Chrome will *tend* to have less tabs impacted by
one leaking, though the leaking tab will still be ugly.

Sometimes the leak manifests simply as a leak (like the washington post
tab I just killed and got 3.5 of the 5GB in use by a content process
back).  Others (depending I assume on what is leaked and how it's kept
alive) cause a core (each) to be chewed doing continual GC/CC (or
processing events in app JS touching some insanely-long internal
structure), slowing the process (and system) to a crawl.


*Generally* these are caused by leaving a page up for a while - navigate
and leave, and you never notice it (part of why website developers don't
fix these, or perhaps don't care (much)).  But even walking away from a
machine with one or a couple of tabs, and coming back the next day can
present you with an unusable tab/browser, or a "this tab has crashed"
screen.


Hoping site owners (or worse, ad producers) will fix their leaks is a
losing game, though we should encourage it and offer tools where
possible.  However, we need a broader solution (or at least tools) for
dealing with this for users.


I don't have a magic-bullet solution, and I'm sure there isn't one given
JS and GC as a core of browsers.  I do think we need to talk about it
(and perhaps not just Mozilla), and find some way to help users here.


One half-baked idea is to provide a UI intervention similar to what we
do on JS that runs too-long, and ask the user if they want to freeze the
tab (or better yet in this case, reload it).  If we trigger before the
side-effects get out of hand, freezing may be ok.  We should also
identify the tab (in the message/popup, and visually in the tab bar - we
don't do either today).  This solution has issues (though freezing tabs
now that "slow down your browser" does too, but it's still useful
overall).

We can also look at tools to characterize site leaks or identify
patterns that point to a leak before it gets out of hand.  Also tools
that help website builders identify if their pages are leaking in the
field (memory-use triggers? within-tab about:memory dumps available to
the page?)

Re: Website memory leaks

[Randell Jesup]

>Many of the pages causing these leaks are major sites, like nytimes.com,
>washington post, cnn, arstechnica, Atlantic, New Yorker, etc.
...
>Perhaps we can also push to limit memory use (CPU use??) in embedded
>ads/restricted-iframes/etc, so sites can stop ads from destroying the
>website performance for users over time.  I've often seen ads taking
>300MB-2GB.

So in support of this general concept of limiting ad memory use:

https://www.scientificamerican.com/article/should-iconic-lake-powell-be-drained/
doesn't leak if you load it -- until you scroll down, and the ads load.
Then it leaks forever... to the tune of 1GB leaked in 5-10 minutes.
Differential about:memory reports show that what's primarily leaking are
ads, in particular at this moment:

400.25 MB (43.45%) -- detached
  398.81 MB (43.30%) -- 
window(https://tpc.googlesyndication.com/safeframe/1-0-13/html/container.html)
seems like the worst culprit, plus
406.69 MB (44.15%) --  
top(https://www.scientificamerican.com/article/should-iconic-lake-powell-be-drained/,id=NNN)
  327.79 MB (35.59%) -- js-zone(0xNNN)

Other ads have leaked a few MB to 75MB each.

Also, as soon as I scrolled down CPU use went from ~0% for the process
to ~20% (on a 4-thread/core AMD CPU).

Worse yet (perhaps bug 1410381??) when I hit reload CPU use dropped (not
to 0).  10 sec later, Mem use climbed 3.7GB to 4.5GB.  then dropped to
3.7GB and climbed back to 4.5GB.  Then dropped to 2.7GB - barely above
where it was before I scrolled down - and stayed stable. (Note: I hit
reload with the current position near the bottom with 1 ad visible (no
video).

Using an additional GB+ of memory in order to free 1GB of memory
seems... excessive.

-- 
Randell Jesup, Mozilla Corp
remove "news" for personal email
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Website memory leaks

[David Durst]

What you're describing having seen is, I think, exactly what I've been
trying to reproduce in a now-blocking-57 bug (1398652).

By your description, the only thing that makes sense to me -- to account
for unknown/unknowable changes on the site -- is to track potential runaway
growth of the content process. I realize how stupid this sounds, but what I
observed was that when the content process was in trouble, it continued to
grow. And the effect/impact builds: first the tab, then the process, then
the browser, then the OS. So if there's a way to determine that "this
process has only grown in the past X amount of time" or past a certain
threshold, that's the best indicator I've come up with.

I don't know what we'd do at that point -- force killing the content
process sounds severe (though possibly correct) -- or some alert similar to
the dreaded slow script one.


--
David Durst [:ddurst]

On Thu, Nov 2, 2017 at 11:31 AM, Randell Jesup 
wrote:

> >about:performance can provide the tab/pid mapping, with some performance
> >indexes.
> >This might help solve your issue listed in the side note.
>
> mconley told me in IRC that today's nightly has brought back the PID in
> the tooltip (in Nightly only); it was accidentally removed.
>
> about:performance can be useful, but 3 tabs vs all tabs is too coarse
> for me, and things like "site leaked memory and is slowing CC" I presume
> doesn't show up in the 'heat' for the tab there.
>
> --
> Randell Jesup, Mozilla Corp
> remove "news" for personal email
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Website memory leaks

[smaug]

This has been an issue forever, and there aren't really good tools on any 
browser, as far as
I know, for web devs to debug their leaks.
Internally we do have useful data (CC and GC graphs and such), but would need 
quite some ux skills to design some good
UI to deal with leaks. Also, the data to deal with is often massive, so the 
tool should be implemented that in mind.

Ads not destroying the performance is a bit different issue in general, and is
somewhat part of the Quantum DOM and iframe throttling. We don't have all of 
that enabled yet.
But sure, ads using too much memory is something to think some more.

We do have pids in e10s-Nightly tab tooltips. There was just a regression for 
couple of days.

On 11/02/2017 04:34 PM, Randell Jesup wrote:

[Note: I'm a tab-hoarder - but that doesn't really cause this problem]

tl;dr: we should look at something (roughly) like the existing "page is
making your browser slow" dialog for website leaks.


Over the last several months (and maybe the last year), I've noticed an
increasing problem in websites: runaway leaks, leading to
a poorly-performing browser or OOM crashes.  Yes, these have *always*
occurred (people write bad code, shock!), but the incidence seems to
have gotten much worse 'recently' (anecdotally).

This may be the result of ads/ad networks (in some cases it provably
is), in other cases it's the sites themselves (like the 4M copies of
":DIV" that facebook was creating over a day if you left a message
visible, plus GB of related leaks (80K copies of many strings/objects).
Many of the pages causing these leaks are major sites, like nytimes.com,
washington post, cnn, arstechnica, Atlantic, New Yorker, etc.

However, regardless of who's making the error or the details, I've been
noticing that I'm having to "hunt for the bad tab" more and more
frequently (usually via about:memory, which users wouldn't
do/understand).  Multi-e10s helps a bit and some tabs won't degrade, but
also enables my system to get into phyical-memory-pressure faster.  (The
machine I notice this on is Win10, 12GB ram, quad-core AMD 8xxx (older,
but 3.5GHz).  I'm running Nightly (32-bit) with one profile (for
facebook), and beta (64bit) with another profile).

While physical-memory pressure causes problems (including for the system
as a whole), the leaking can make Content processes unresponsive, to the
point where about:memory doesn't get data for them (or it's incomplete).
This makes it hard-to-impossible to fix the problem; I kill Content
processes by hand in that case - regular users would just force-close the
browser (and perhaps start Chrome instead of restarting.)  We see an
insanely high number of OOMs in the field; likely this is a major
contributing factor.  Chrome will *tend* to have less tabs impacted by
one leaking, though the leaking tab will still be ugly.

Sometimes the leak manifests simply as a leak (like the washington post
tab I just killed and got 3.5 of the 5GB in use by a content process
back).  Others (depending I assume on what is leaked and how it's kept
alive) cause a core (each) to be chewed doing continual GC/CC (or
processing events in app JS touching some insanely-long internal
structure), slowing the process (and system) to a crawl.


*Generally* these are caused by leaving a page up for a while - navigate
and leave, and you never notice it (part of why website developers don't
fix these, or perhaps don't care (much)).  But even walking away from a
machine with one or a couple of tabs, and coming back the next day can
present you with an unusable tab/browser, or a "this tab has crashed"
screen.


Hoping site owners (or worse, ad producers) will fix their leaks is a
losing game, though we should encourage it and offer tools where
possible.  However, we need a broader solution (or at least tools) for
dealing with this for users.


I don't have a magic-bullet solution, and I'm sure there isn't one given
JS and GC as a core of browsers.  I do think we need to talk about it
(and perhaps not just Mozilla), and find some way to help users here.


One half-baked idea is to provide a UI intervention similar to what we
do on JS that runs too-long, and ask the user if they want to freeze the
tab (or better yet in this case, reload it).  If we trigger before the
side-effects get out of hand, freezing may be ok.  We should also
identify the tab (in the message/popup, and visually in the tab bar - we
don't do either today).  This solution has issues (though freezing tabs
now that "slow down your browser" does too, but it's still useful
overall).

We can also look at tools to characterize site leaks or identify
patterns that point to a leak before it gets out of hand.  Also tools
that help website builders identify if their pages are leaking in the
field (memory-use triggers? within-tab about:memory dumps available to
the page?)

Perhaps we can also push to limit memory use (CPU use??) in embedded
ads/restricted-iframes/etc, so sites can stop ads from 

Re: Website memory leaks

[Randell Jesup]

>about:performance can provide the tab/pid mapping, with some performance
>indexes.
>This might help solve your issue listed in the side note.

mconley told me in IRC that today's nightly has brought back the PID in
the tooltip (in Nightly only); it was accidentally removed.

about:performance can be useful, but 3 tabs vs all tabs is too coarse
for me, and things like "site leaked memory and is slowing CC" I presume
doesn't show up in the 'heat' for the tab there.

-- 
Randell Jesup, Mozilla Corp
remove "news" for personal email
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Website memory leaks

[Shih-Chiang Chien]

about:performance can provide the tab/pid mapping, with some performance
indexes.
This might help solve your issue listed in the side note.

Best Regards,
Shih-Chiang Chien
Mozilla Taiwan

On Thu, Nov 2, 2017 at 3:34 PM, Randell Jesup  wrote:

> [Note: I'm a tab-hoarder - but that doesn't really cause this problem]
>
> tl;dr: we should look at something (roughly) like the existing "page is
> making your browser slow" dialog for website leaks.
>
>
> Over the last several months (and maybe the last year), I've noticed an
> increasing problem in websites: runaway leaks, leading to
> a poorly-performing browser or OOM crashes.  Yes, these have *always*
> occurred (people write bad code, shock!), but the incidence seems to
> have gotten much worse 'recently' (anecdotally).
>
> This may be the result of ads/ad networks (in some cases it provably
> is), in other cases it's the sites themselves (like the 4M copies of
> ":DIV" that facebook was creating over a day if you left a message
> visible, plus GB of related leaks (80K copies of many strings/objects).
> Many of the pages causing these leaks are major sites, like nytimes.com,
> washington post, cnn, arstechnica, Atlantic, New Yorker, etc.
>
> However, regardless of who's making the error or the details, I've been
> noticing that I'm having to "hunt for the bad tab" more and more
> frequently (usually via about:memory, which users wouldn't
> do/understand).  Multi-e10s helps a bit and some tabs won't degrade, but
> also enables my system to get into phyical-memory-pressure faster.  (The
> machine I notice this on is Win10, 12GB ram, quad-core AMD 8xxx (older,
> but 3.5GHz).  I'm running Nightly (32-bit) with one profile (for
> facebook), and beta (64bit) with another profile).
>
> While physical-memory pressure causes problems (including for the system
> as a whole), the leaking can make Content processes unresponsive, to the
> point where about:memory doesn't get data for them (or it's incomplete).
> This makes it hard-to-impossible to fix the problem; I kill Content
> processes by hand in that case - regular users would just force-close the
> browser (and perhaps start Chrome instead of restarting.)  We see an
> insanely high number of OOMs in the field; likely this is a major
> contributing factor.  Chrome will *tend* to have less tabs impacted by
> one leaking, though the leaking tab will still be ugly.
>
> Sometimes the leak manifests simply as a leak (like the washington post
> tab I just killed and got 3.5 of the 5GB in use by a content process
> back).  Others (depending I assume on what is leaked and how it's kept
> alive) cause a core (each) to be chewed doing continual GC/CC (or
> processing events in app JS touching some insanely-long internal
> structure), slowing the process (and system) to a crawl.
>
>
> *Generally* these are caused by leaving a page up for a while - navigate
> and leave, and you never notice it (part of why website developers don't
> fix these, or perhaps don't care (much)).  But even walking away from a
> machine with one or a couple of tabs, and coming back the next day can
> present you with an unusable tab/browser, or a "this tab has crashed"
> screen.
>
>
> Hoping site owners (or worse, ad producers) will fix their leaks is a
> losing game, though we should encourage it and offer tools where
> possible.  However, we need a broader solution (or at least tools) for
> dealing with this for users.
>
>
> I don't have a magic-bullet solution, and I'm sure there isn't one given
> JS and GC as a core of browsers.  I do think we need to talk about it
> (and perhaps not just Mozilla), and find some way to help users here.
>
>
> One half-baked idea is to provide a UI intervention similar to what we
> do on JS that runs too-long, and ask the user if they want to freeze the
> tab (or better yet in this case, reload it).  If we trigger before the
> side-effects get out of hand, freezing may be ok.  We should also
> identify the tab (in the message/popup, and visually in the tab bar - we
> don't do either today).  This solution has issues (though freezing tabs
> now that "slow down your browser" does too, but it's still useful
> overall).
>
> We can also look at tools to characterize site leaks or identify
> patterns that point to a leak before it gets out of hand.  Also tools
> that help website builders identify if their pages are leaking in the
> field (memory-use triggers? within-tab about:memory dumps available to
> the page?)
>
> Perhaps we can also push to limit memory use (CPU use??) in embedded
> ads/restricted-iframes/etc, so sites can stop ads from destroying the
> website performance for users over time.  I've often seen ads taking
> 300MB-2GB.
>
> I'm sure there are better/more ideas - suggestions?  What thought has
> gone on in the past? (this *can't* be a new observation or idea)
>
>
>
> Also, I suspect that website behaviors/leaks like this are part of why
> many users have developed a habit of 

Website memory leaks

[Randell Jesup]

[Note: I'm a tab-hoarder - but that doesn't really cause this problem]

tl;dr: we should look at something (roughly) like the existing "page is
making your browser slow" dialog for website leaks.


Over the last several months (and maybe the last year), I've noticed an
increasing problem in websites: runaway leaks, leading to
a poorly-performing browser or OOM crashes.  Yes, these have *always*
occurred (people write bad code, shock!), but the incidence seems to
have gotten much worse 'recently' (anecdotally).

This may be the result of ads/ad networks (in some cases it provably
is), in other cases it's the sites themselves (like the 4M copies of
":DIV" that facebook was creating over a day if you left a message
visible, plus GB of related leaks (80K copies of many strings/objects).
Many of the pages causing these leaks are major sites, like nytimes.com,
washington post, cnn, arstechnica, Atlantic, New Yorker, etc.

However, regardless of who's making the error or the details, I've been
noticing that I'm having to "hunt for the bad tab" more and more
frequently (usually via about:memory, which users wouldn't
do/understand).  Multi-e10s helps a bit and some tabs won't degrade, but
also enables my system to get into phyical-memory-pressure faster.  (The
machine I notice this on is Win10, 12GB ram, quad-core AMD 8xxx (older,
but 3.5GHz).  I'm running Nightly (32-bit) with one profile (for
facebook), and beta (64bit) with another profile).

While physical-memory pressure causes problems (including for the system
as a whole), the leaking can make Content processes unresponsive, to the
point where about:memory doesn't get data for them (or it's incomplete).
This makes it hard-to-impossible to fix the problem; I kill Content
processes by hand in that case - regular users would just force-close the
browser (and perhaps start Chrome instead of restarting.)  We see an
insanely high number of OOMs in the field; likely this is a major
contributing factor.  Chrome will *tend* to have less tabs impacted by
one leaking, though the leaking tab will still be ugly.

Sometimes the leak manifests simply as a leak (like the washington post
tab I just killed and got 3.5 of the 5GB in use by a content process
back).  Others (depending I assume on what is leaked and how it's kept
alive) cause a core (each) to be chewed doing continual GC/CC (or
processing events in app JS touching some insanely-long internal
structure), slowing the process (and system) to a crawl.


*Generally* these are caused by leaving a page up for a while - navigate
and leave, and you never notice it (part of why website developers don't
fix these, or perhaps don't care (much)).  But even walking away from a
machine with one or a couple of tabs, and coming back the next day can
present you with an unusable tab/browser, or a "this tab has crashed"
screen.


Hoping site owners (or worse, ad producers) will fix their leaks is a
losing game, though we should encourage it and offer tools where
possible.  However, we need a broader solution (or at least tools) for
dealing with this for users.


I don't have a magic-bullet solution, and I'm sure there isn't one given
JS and GC as a core of browsers.  I do think we need to talk about it
(and perhaps not just Mozilla), and find some way to help users here.


One half-baked idea is to provide a UI intervention similar to what we
do on JS that runs too-long, and ask the user if they want to freeze the
tab (or better yet in this case, reload it).  If we trigger before the
side-effects get out of hand, freezing may be ok.  We should also
identify the tab (in the message/popup, and visually in the tab bar - we
don't do either today).  This solution has issues (though freezing tabs
now that "slow down your browser" does too, but it's still useful
overall).

We can also look at tools to characterize site leaks or identify
patterns that point to a leak before it gets out of hand.  Also tools
that help website builders identify if their pages are leaking in the
field (memory-use triggers? within-tab about:memory dumps available to
the page?)  

Perhaps we can also push to limit memory use (CPU use??) in embedded
ads/restricted-iframes/etc, so sites can stop ads from destroying the
website performance for users over time.  I've often seen ads taking
300MB-2GB.

I'm sure there are better/more ideas - suggestions?  What thought has
gone on in the past? (this *can't* be a new observation or idea)



Also, I suspect that website behaviors/leaks like this are part of why
many users have developed a habit of browsing and closing, opening few
tabs at any point in a session (since leaving them open *sometimes*
hurts you a lot).

Part of why I care about this is that I'm looking at how far we can go
with multiple content processes, site isolation, and moving (more)
security-sensitive and/or crashy services into separate processes -- all
of which come into play here.

Side-note: removing the process-id from tabs in mult-e10s has