Re: We live in a memory-constrained world
On Tue, Mar 04, 2014 at 09:48:33AM +0200, Henri Sivonen wrote: On Fri, Feb 28, 2014 at 8:09 PM, L. David Baron dba...@dbaron.org wrote: In other words, whenever you have a pointer in a static data structure pointing to some other data, that pointer needs to get fixed up when the library loads, which makes the memory that pointer is in less likely to be shared across processes (depending, I guess, on how many processes are able to load the library at its default address, which may in turn depend on security features that try to randomize library base addresses). This also slows down loading of shared libraries. So all things considered, do we want things like static atoms and the HTML parser's pre-interned element name and attribute name objects (which have pointers to static atoms and a virtual method) to move from the heap to POD-like syntax even if it results in relocations or, with MSVC, static initializers? its generally gcc that does dumb things with static initializers, but that can generally be fixed with liberal use of constexpr. Anyway I suspect the real answer is it's complicated ;) but it's probably a good idea at least for things on the staart up path anyway adding a relocation and saving a call to malloc before we can parse html is probably a win on its own. Shouldn't be an issue with Nuwa-cloned processes on B2G, though. Are static atoms and the HTML parser's pre-interned element name and attribute name objects that are on the heap shared between processes under Nuwa already? I.e. is the heap cloned with copy-on-write sharing? On the memory page granularity, right? Do we know if the aiui yes the heap is made copy on write at the time fork(2) is called to create the Nuwa process, and presumably we don't have KSM turned on so only the initial heap will ever be shared. I'd guess that HTML parser stuff you mentioned is created beforewe fork the Nuwa process and so its included. stuff we heap-allocate at startup pack nicely into memory pages so that they won't have free spots that the allocator would use after cloning? probably not perfectly, but it seems like in practice it does fairly well other wise Nuwa wouldn't help. Trev -- Henri Sivonen hsivo...@hsivonen.fi https://hsivonen.fi/ ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform signature.asc Description: Digital signature ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: We live in a memory-constrained world
On Mon, Mar 3, 2014 at 11:48 PM, Henri Sivonen hsivo...@hsivonen.fi wrote: Are static atoms and the HTML parser's pre-interned element name and attribute name objects that are on the heap shared between processes under Nuwa already? I.e. is the heap cloned with copy-on-write sharing? On the memory page granularity, right? Do we know if the stuff we heap-allocate at startup pack nicely into memory pages so that they won't have free spots that the allocator would use after cloning? https://bugzilla.mozilla.org/show_bug.cgi?id=948648 is open for investigating this. Preliminary results seem to indicate that things work pretty well at the moment -- i.e. not much sharing is lost -- but I may be misinterpreting. Nick ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: We live in a memory-constrained world
On Fri, Feb 28, 2014 at 8:09 PM, L. David Baron dba...@dbaron.org wrote: In other words, whenever you have a pointer in a static data structure pointing to some other data, that pointer needs to get fixed up when the library loads, which makes the memory that pointer is in less likely to be shared across processes (depending, I guess, on how many processes are able to load the library at its default address, which may in turn depend on security features that try to randomize library base addresses). This also slows down loading of shared libraries. So all things considered, do we want things like static atoms and the HTML parser's pre-interned element name and attribute name objects (which have pointers to static atoms and a virtual method) to move from the heap to POD-like syntax even if it results in relocations or, with MSVC, static initializers? Shouldn't be an issue with Nuwa-cloned processes on B2G, though. Are static atoms and the HTML parser's pre-interned element name and attribute name objects that are on the heap shared between processes under Nuwa already? I.e. is the heap cloned with copy-on-write sharing? On the memory page granularity, right? Do we know if the stuff we heap-allocate at startup pack nicely into memory pages so that they won't have free spots that the allocator would use after cloning? -- Henri Sivonen hsivo...@hsivonen.fi https://hsivonen.fi/ ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: We live in a memory-constrained world
Henri Sivonen wrote: Any chance static atoms could reside as plain old static C data structures in the data segment of libxul.so instead of being heap-allocated? At least under MSVC, they have vtables, so they need to be constructed, so they're not static. Note that their string storage is static, through the use of fake string buffers (possibly this should be switched to fake literal strings). -- Warning: May contain traces of nuts. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: We live in a memory-constrained world
On Fri, Feb 28, 2014 at 1:04 PM, Neil n...@parkwaycc.co.uk wrote: At least under MSVC, they have vtables, so they need to be constructed, so they're not static. So structs that inherit at least one virtual method can't be plain old C data? That surprises me. And we still don't want to give the dynamic linker initializer code to run, right? -- Henri Sivonen hsivo...@hsivonen.fi https://hsivonen.fi/ ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: We live in a memory-constrained world
http://en.wikipedia.org/wiki/Plain_Old_Data_Structures confirms that POD can't have a vptr :-) Benoit 2014-02-28 7:39 GMT-05:00 Henri Sivonen hsivo...@hsivonen.fi: On Fri, Feb 28, 2014 at 1:04 PM, Neil n...@parkwaycc.co.uk wrote: At least under MSVC, they have vtables, so they need to be constructed, so they're not static. So structs that inherit at least one virtual method can't be plain old C data? That surprises me. And we still don't want to give the dynamic linker initializer code to run, right? -- Henri Sivonen hsivo...@hsivonen.fi https://hsivonen.fi/ ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: We live in a memory-constrained world
- Original Message -
http://en.wikipedia.org/wiki/Plain_Old_Data_Structures
confirms that POD can't have a vptr :-)
Just because it's not POD in the language doesn't mean that it necessarily
requires a static initializer in all implementations:
class Interface {
public:
constexpr Interface() {}
virtual int f() = 0;
};
class Concrete : public Interface {
public:
constexpr Concrete() {}
virtual int f();
};
Concrete c;
run through |g++ -std=c++0x -O2 -o - -S| (gcc 4.6, x86-64) gives:
.file vtables.cpp
.globl c
.data
.align 8
.type c, @object
.size c, 8
c:
.quad _ZTV8Concrete+16
No initialization code in sight. clang does the same, with or without the
constexpr.
-Nathan
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
Re: We live in a memory-constrained world
On Fri, Feb 28, 2014 at 05:59:13AM -0800, Nathan Froyd wrote:
- Original Message -
http://en.wikipedia.org/wiki/Plain_Old_Data_Structures
confirms that POD can't have a vptr :-)
class Interface {
public:
constexpr Interface() {}
virtual int f() = 0;
};
class Concrete : public Interface {
public:
constexpr Concrete() {}
virtual int f();
};
Concrete c;
run through |g++ -std=c++0x -O2 -o - -S| (gcc 4.6, x86-64) gives:
.file vtables.cpp
.globl c
.data
.align 8
.type c, @object
.size c, 8
c:
.quad _ZTV8Concrete+16
No initialization code in sight. clang does the same, with or without the
constexpr.
I think the actually tricky part of making the atoms themselves live in
rodata is that they need to be layed out as if they were in a hash table
so that we can get the atom for a string in O(1). I think we could do
that by reimplementing some bits of the hash table in python and then
spewing out a giant .S file layed out as if its a hash table, but seems
like a bit of work and its not clear off hand to me that it would be a
big win.
Trev
-Nathan
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
Re: We live in a memory-constrained world
Neil wrote: Henri Sivonen wrote: Any chance static atoms could reside as plain old static C data structures in the data segment of libxul.so instead of being heap-allocated? At least under MSVC, they have vtables, so they need to be constructed, so they're not static. Note that their string storage is static, through the use of fake string buffers (possibly this should be switched to fake literal strings). Perhaps we could probably use string buffers directly as atoms. Static atoms would be replaced by their underlying fake string buffers (perhaps identified by a zero refcount?) while other atoms would be refcounted buffers typically from their original nsString objects. -- Warning: May contain traces of nuts. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: We live in a memory-constrained world
On 2/28/14 12:26 PM, Neil wrote: Perhaps we could probably use string buffers directly as atoms. Atoms have various info string buffers don't (like the hashcode). -Boris ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: We live in a memory-constrained world
On Friday 2014-02-28 11:04 +, Neil wrote: Henri Sivonen wrote: Any chance static atoms could reside as plain old static C data structures in the data segment of libxul.so instead of being heap-allocated? At least under MSVC, they have vtables, so they need to be constructed, so they're not static. Note that their string storage is static, through the use of fake string buffers (possibly this should be switched to fake literal strings). Remember that since this is all in a shared library (which might be loaded at different addresses, or even different addresses in different processes [1]), you also have to worry about relocations. In other words, whenever you have a pointer in a static data structure pointing to some other data, that pointer needs to get fixed up when the library loads, which makes the memory that pointer is in less likely to be shared across processes (depending, I guess, on how many processes are able to load the library at its default address, which may in turn depend on security features that try to randomize library base addresses). This also slows down loading of shared libraries. Or something like that; please correct me if this is wrong. -David [1] Shouldn't be an issue with Nuwa-cloned processes on B2G, though. -- 턞 L. David Baron http://dbaron.org/ 턂 턢 Mozilla https://www.mozilla.org/ 턂 Before I built a wall I'd ask to know What I was walling in or walling out, And to whom I was like to give offense. - Robert Frost, Mending Wall (1914) signature.asc Description: Digital signature ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: We live in a memory-constrained world
On Fri, Feb 21, 2014 at 11:38 PM, Nicholas Nethercote n.netherc...@gmail.com wrote: We now live in a memory-constrained world. By we, I mean anyone working on Mozilla platform code. I realize that .so size matters much less than per-process heap-allocated stuff, but still: On the i18n front, there are binary size gains to be had by not building parts of ICU that we don't use. See https://bugzilla.mozilla.org/show_bug.cgi?id=944348 . (Before someone suggests using the ICU encoding converters: 1) We are building ICU converters that MUST NOT be exposed to the Web platform and we have no internal use for [e.g. BOCU-1] and 2) unless upstream ICU decides they want to track the Encoding Standard, replacing our converters with theirs may easily regress our Encoding Standard compliance, i.e. Web compatibility.) Also, in our own i18n code, there's a whole bunch of stuff that's dead code in Firefox (thanks to Encoding Standard work) but exposed in Thunderbird. The main problems are: 1) Gecko developers aren't really supposed to use time for Thunderbird development but unless comm-central burns, there's no sense of urgency from the Thunderdbird side to move stuff over, but making comm-central burn would be not be a polite way to make progress. 2) The Thunderbird developers aren't sure which bits they want to keep. Before ESR goes to release and people complain, no one *really* knows what needs to stay in TB. 3) There doesn't exist (see #1) a comm-central XPCOM module that could naturally receive (copy and paste) code from the m-c nsUConvModule. See https://bugzilla.mozilla.org/show_bug.cgi?id=937056#c3 An for something entirely different: Any chance static atoms could reside as plain old static C data structures in the data segment of libxul.so instead of being heap-allocated? -- Henri Sivonen hsivo...@hsivonen.fi https://hsivonen.fi/ ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: We live in a memory-constrained world
On Tue, Feb 25, 2014 at 8:18 PM, Mike Hommey m...@glandium.org wrote: I never understood why we need those jobs to be builds. Why not turn --enable-valgrind on m-c builds, and run valgrind as a test job? --disable-jemalloc is needed as well. That shouldn't be needed anymore with --soname-synonyms=somalloc=NONE on the valgrind command line. I did not know about that! Thank you. I filed https://bugzilla.mozilla.org/show_bug.cgi?id=977067 to update |mach valgrind-test| accordingly, and I will also update the docs. That will make it easier to use standard builds... turning on --enable-valgrind for m-c builds is pretty reasonable, because it makes only tiny differences (insertion of a few nops in non-perf-critical places) to the resulting code. Nick ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: We live in a memory-constrained world
On 2014-02-26, 8:44 AM, Nicholas Nethercote wrote: On Tue, Feb 25, 2014 at 8:18 PM, Mike Hommey m...@glandium.org wrote: I never understood why we need those jobs to be builds. Why not turn --enable-valgrind on m-c builds, and run valgrind as a test job? --disable-jemalloc is needed as well. That shouldn't be needed anymore with --soname-synonyms=somalloc=NONE on the valgrind command line. I did not know about that! Thank you. I filed https://bugzilla.mozilla.org/show_bug.cgi?id=977067 to update |mach valgrind-test| accordingly, and I will also update the docs. That will make it easier to use standard builds... turning on --enable-valgrind for m-c builds is pretty reasonable, because it makes only tiny differences (insertion of a few nops in non-perf-critical places) to the resulting code. Sorry if this is driving the thread off-topic, but can you please provide a list of the things that --enable-valgrind changes? I am very curious to know the implications of turning this on. Thanks! Ehsan ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: We live in a memory-constrained world
Splitting the valgrind tests up and running them separately as test jobs in TBPL is definitely something the A*Team can help with. I've filed bug 977240 for this. Jonathan On 2/25/14 7:25 PM, Nicholas Nethercote wrote: On Tue, Feb 25, 2014 at 2:32 PM, Mike Hommey m...@glandium.org wrote: I never understood why we need those jobs to be builds. Why not turn --enable-valgrind on m-c builds, and run valgrind as a test job? --disable-jemalloc is needed as well. As for the structure... I just took what already existed and got it into good enough shape to make visible on TBPL. (That was more than enough for a non-TBPL/buildbot expert like me to take on.) I'm fine with the idea of the Valgrind test job being split up into suites and using the test machines for the testing part, but I'm not jumping up and down to volunteer to do it. Nick ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: We live in a memory-constrained world
In valgrind on FxOS devices (right now only nexus 4/5), we run checks against IPC. Unfortunately there's no automation there at the moment. We've got a goal to get IPC turned on for b2g desktop ASAP though, at which point we might be able to look at that. - Original Message - From: Nicholas Nethercote n.netherc...@gmail.com To: Ehsan Akhgari ehsan.akhg...@gmail.com Cc: dev-platform dev-platform@lists.mozilla.org Sent: Monday, February 24, 2014 7:08:31 PM Subject: Re: We live in a memory-constrained world On Mon, Feb 24, 2014 at 6:50 PM, Ehsan Akhgari ehsan.akhg...@gmail.com wrote: Nick, do we run any kind of automated leak checking through valgrind? I found bug 976363 today, it's something that any automated leak detection tool should be able to catch. The Valgrind test job does leak checking, and it's recently caught some leaks and caused the offending patches to be backed out. However, the test coverage is pretty meagre, and it's desktop only so can't detect leaks in IPC code. I believe the ASan builds also do leak checking. Their coverage is much better, but still doesn't cover IPC. Nick ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: We live in a memory-constrained world
On 2/24/2014, 10:08 PM, Nicholas Nethercote wrote: On Mon, Feb 24, 2014 at 6:50 PM, Ehsan Akhgari ehsan.akhg...@gmail.com wrote: Nick, do we run any kind of automated leak checking through valgrind? I found bug 976363 today, it's something that any automated leak detection tool should be able to catch. The Valgrind test job does leak checking, and it's recently caught some leaks and caused the offending patches to be backed out. However, the test coverage is pretty meagre, and it's desktop only so can't detect leaks in IPC code. These leaks are reproducible on desktop, as part of mochitest-2. For reproducing them locally, you need to run the tests in dom/browser-element. Any idea why they were not caught then? I believe the ASan builds also do leak checking. Their coverage is much better, but still doesn't cover IPC. I don't think that they do any kind of leak checking. (Also, we run quite a bit of the IPC code on desktop for things like plugins, thumbnail generation for about:newtab and mochitests that specifically test out of process frames.) Cheers, Ehsan ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: We live in a memory-constrained world
On Tue, Feb 25, 2014 at 12:48 PM, Ehsan Akhgari ehsan.akhg...@gmail.com wrote: The Valgrind test job does leak checking, and it's recently caught some leaks and caused the offending patches to be backed out. However, the test coverage is pretty meagre, and it's desktop only so can't detect leaks in IPC code. These leaks are reproducible on desktop, as part of mochitest-2. For reproducing them locally, you need to run the tests in dom/browser-element. Any idea why they were not caught then? Valgrind doesn't run mochitests because it's far too slow. It currently just runs the PGO profiling code, which is in build/pgo/index.html. It would be good to increase the coverage -- separating the Valgrind test code from the PGO profiling code will be easy -- but we can't do it too much otherwise the job will be too slow. Currently it typically takes ~45 minutes and ~38 minutes of that is build time. Nick ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: We live in a memory-constrained world
On 2014-02-25, 3:53 PM, Nicholas Nethercote wrote: On Tue, Feb 25, 2014 at 12:48 PM, Ehsan Akhgari ehsan.akhg...@gmail.com wrote: The Valgrind test job does leak checking, and it's recently caught some leaks and caused the offending patches to be backed out. However, the test coverage is pretty meagre, and it's desktop only so can't detect leaks in IPC code. These leaks are reproducible on desktop, as part of mochitest-2. For reproducing them locally, you need to run the tests in dom/browser-element. Any idea why they were not caught then? Valgrind doesn't run mochitests because it's far too slow. It currently just runs the PGO profiling code, which is in build/pgo/index.html. It would be good to increase the coverage -- separating the Valgrind test code from the PGO profiling code will be easy -- but we can't do it too much otherwise the job will be too slow. Currently it typically takes ~45 minutes and ~38 minutes of that is build time. OK this explains it very well, thanks! How much slower is a valgrind run of let's say mochitest-1 on a test slave? I don't think that we can run many interesting tests on valgrind on the build machine (and the PGO test set is *really* old and doesn't cover a lot of the platform features.) Thanks! Ehsan ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: We live in a memory-constrained world
On Tue, Feb 25, 2014 at 1:10 PM, Ehsan Akhgari ehsan.akhg...@gmail.com wrote: How much slower is a valgrind run of let's say mochitest-1 on a test slave? I don't think that we can run many interesting tests on valgrind on the build machine (and the PGO test set is *really* old and doesn't cover a lot of the platform features.) I haven't measured mochitests, but Valgrind's slowdown is usually something like 20x. I know Julian Seward occasionally does full mochitest runs under Valgrind and it takes something like 12 hours. Nick ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: We live in a memory-constrained world
On Tue, Feb 25, 2014 at 2:32 PM, Mike Hommey m...@glandium.org wrote: I never understood why we need those jobs to be builds. Why not turn --enable-valgrind on m-c builds, and run valgrind as a test job? --disable-jemalloc is needed as well. As for the structure... I just took what already existed and got it into good enough shape to make visible on TBPL. (That was more than enough for a non-TBPL/buildbot expert like me to take on.) I'm fine with the idea of the Valgrind test job being split up into suites and using the test machines for the testing part, but I'm not jumping up and down to volunteer to do it. Nick ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: We live in a memory-constrained world
On 2/22/14, 1:39 PM, Nicholas Nethercote wrote: On Sat, Feb 22, 2014 at 11:22 PM, Andreas Gal andreas@gmail.com wrote: So, I'm wondering how much effort we should put in reducing the number of ChromeWorkers. We should continue to use JS in Chrome where it makes sense. Its often easier and faster to write some functionality in JS (and sometimes also safer), and it tends to be more compact when we ship it. In addition to purging caches and stopping idle workers the JS team is also working on making workers (and JS) more memory efficient in general. Naveed might want to chime in here. Brian Hackett's been making great progress in reducing the overhead of workers. https://bugzilla.mozilla.org/show_bug.cgi?id=964059 and https://bugzilla.mozilla.org/show_bug.cgi?id=964057 both landed this week, reducing the overhead of a trivial worker from ~1 MiB to ~0.2 MiB. And https://bugzilla.mozilla.org/show_bug.cgi?id=864927 is the meta-bug, which has ideas for further improvements. Naveed told me that bug 941783 is meta-bug for all B2G DOM worker optimizations, beyond just memory reduction. He is preparing a bug backlog to prioritize those issues. chris ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: We live in a memory-constrained world
On 2/21/2014 5:40 PM, Brian Smith wrote: On Fri, Feb 21, 2014 at 1:38 PM, Nicholas Nethercote n.netherc...@gmail.com wrote: Optimizations that wouldn't have been worthwhile in the desktop-only days are now worthwhile. For example, an optimization that saves 100 KiB of memory per process is pretty worthwhile for Firefox OS. Do you mean 100KiB per child process? How worthwhile is it to cut 100KiB from the parent process? I ask because we have various caches that we use for TLS security (TLS session cache, HSTS, OCSP response cache), and I'd like to know how much memory we can can budge for these features, given that they all affect only a single process (the parent). In some cases, we can throw the information away and re-retrieve and/or recompute it, but only at a cost of reduced security and/or significantly reduced performance. In some cases, we could try to leave the data on disk and avoid loading it into memory, but then we run into main-thread-I/O and/or socket-thread-disk-I/O, both of which are important to avoid for responsiveness/perf reasons. Also, I am concerned that, AFAICT, no TLS-related testing is being done for AWSY. Thus, a lot of my work isn't being measured there. Who is the best person to talk to about getting TLS functionality added to AWSY? Not sure if these are what you are referring to, but I see nss security PORT_Alloc_Util() pretty consistently in parent process DMD. In a recent DMD report I saw it accounting for ~1.2MB over a number of different invocations. Ben ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: We live in a memory-constrained world
On 02/21/2014 02:57 PM, Nicholas Nethercote wrote: On Sat, Feb 22, 2014 at 9:40 AM, Brian Smith br...@briansmith.org wrote: How worthwhile is it to cut 100KiB from the parent process? We don't have a fixed memory budget per se. If it's easy and doesn't have bad side-effects, do it. Otherwise... use your judgment. Having a Firefox OS device to test on helps with evaluating such things. Also, I am concerned that, AFAICT, no TLS-related testing is being done for AWSY. Thus, a lot of my work isn't being measured there. Who is the best person to talk to about getting TLS functionality added to AWSY? John Schoenick maintains AWSY. AWSY's current test is dead simple, and doesn't test many modern features: https://github.com/Nephyrin/MozAreWeSlimYet/blob/master/mozmill_endurance_test/test1.js I don't have the time or probably qualifications to design something better than TP5 -- but if someone wants to tackle making a better set of test pages, I think we'd gladly use it on AWSY. Ideally we'd be replaying network sessions of things like browsing a page, interacting with gmail, etc.. Nick ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: We live in a memory-constrained world
Nick, do we run any kind of automated leak checking through valgrind? I found bug 976363 today, it's something that any automated leak detection tool should be able to catch. Thanks! Ehsan On 2/21/2014, 4:38 PM, Nicholas Nethercote wrote: Greetings, We now live in a memory-constrained world. By we, I mean anyone working on Mozilla platform code. When desktop Firefox was our only product, this wasn't especially true -- bad leaks and the like were a problem, sure, but ordinary usage wasn't much of an issue. But now with Firefox on Android and particularly Firefox OS, it is most definitely true. In particular, work is currently underway to get Firefox OS working on devices that only have 128 MiB of RAM. The codename for these devices is Tarako (https://wiki.mozilla.org/FirefoxOS/Tarako). In case it's not obvious, the memory situation on these devices is *tight*. Optimizations that wouldn't have been worthwhile in the desktop-only days are now worthwhile. For example, an optimization that saves 100 KiB of memory per process is pretty worthwhile for Firefox OS. On a phone, memory consumption can be the difference between something working and something not working in a much more clear-cut way than is typical on desktop. Conversely, landing new features that use extra memory need to be considered carefully. If you're planning a new feature and it will only use a few MiB on Firefox OS, think again. We probably cannot afford it. https://bugzilla.mozilla.org/show_bug.cgi?id=975367 is the bug that motivated me to write this email, but I'm also reminded of Firefox 4, which was a release with very high memory consumption partly because several groups independently made decisions that using extra memory was ok for a particular sub-system and the combined effect was bad. We live in a memory-constrained world. Nick ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: We live in a memory-constrained world
On Mon, Feb 24, 2014 at 6:50 PM, Ehsan Akhgari ehsan.akhg...@gmail.com wrote: Nick, do we run any kind of automated leak checking through valgrind? I found bug 976363 today, it's something that any automated leak detection tool should be able to catch. The Valgrind test job does leak checking, and it's recently caught some leaks and caused the offending patches to be backed out. However, the test coverage is pretty meagre, and it's desktop only so can't detect leaks in IPC code. I believe the ASan builds also do leak checking. Their coverage is much better, but still doesn't cover IPC. Nick ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: We live in a memory-constrained world
- Original Message - The Valgrind test job does leak checking, and it's recently caught some leaks and caused the offending patches to be backed out. However, the test coverage is pretty meagre, and it's desktop only so can't detect leaks in IPC code. I believe the ASan builds also do leak checking. Their coverage is much better, but still doesn't cover IPC. I don't think we run any leak checking with ASan, or at least I've never seen such a report. I filed bug 976414 for investigating LeakSanitizer. Andrew ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: We live in a memory-constrained world
On Fri, Feb 21, 2014 at 9:56 PM, Nicholas Nethercote n.netherc...@gmail.com wrote: If your data is read-only after the point at which Nuwa forks the process (which is once things are mostly loaded but before we load the app) then generally yes. It's copy-on-write at page granularity of course, so if you have sub-page allocations you can't guarantee that it won't be copied. That sounds promising. Is there a way to test whether this is, in fact, happening for the self-hosting compartment? Not that I know of. The OS knows which memory pages are shared, but if it's anonymously mapped data (as the entire malloc heap and GC heaps are) you can only get data for entire segments, which are usually many pages. And knowing which segment(s) the relevant data is in is also difficult. However, Thinker said this: For now, Nuwa is frozen after RecvAppInfo(), from IPC, where PreloadSlowThing() is called. It is started by posting a runnable. So, all runnables posted before Nuwa's one is expected to be run before Nuwa is frozen. XPCOM is initialized before posting Nuwa's runnable. So that should give you a decent idea. It's also worth noting that if you have a mixture of mutable and immutable data, it might be worth trying to segregate them. E.g. if you have a Nuwa-shared page and you modify 1 byte, the entire 4 KiB page will be copied by the modifying content process. (https://bugzilla.mozilla.org/show_bug.cgi?id=948648 is open to look for instances of this.) Nick ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: We live in a memory-constrained world
On Sat, Feb 22, 2014 at 11:22 PM, Andreas Gal andreas@gmail.com wrote: So, I'm wondering how much effort we should put in reducing the number of ChromeWorkers. We should continue to use JS in Chrome where it makes sense. Its often easier and faster to write some functionality in JS (and sometimes also safer), and it tends to be more compact when we ship it. In addition to purging caches and stopping idle workers the JS team is also working on making workers (and JS) more memory efficient in general. Naveed might want to chime in here. Brian Hackett's been making great progress in reducing the overhead of workers. https://bugzilla.mozilla.org/show_bug.cgi?id=964059 and https://bugzilla.mozilla.org/show_bug.cgi?id=964057 both landed this week, reducing the overhead of a trivial worker from ~1 MiB to ~0.2 MiB. And https://bugzilla.mozilla.org/show_bug.cgi?id=864927 is the meta-bug, which has ideas for further improvements. Nick ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: We live in a memory-constrained world
On 02/21/2014 01:38 PM, Nicholas Nethercote wrote: Greetings, We now live in a memory-constrained world. By we, I mean anyone working on Mozilla platform code. When desktop Firefox was our only product, this wasn't especially true -- bad leaks and the like were a problem, sure, but ordinary usage wasn't much of an issue. But now with Firefox on Android and particularly Firefox OS, it is most definitely true. In particular, work is currently underway to get Firefox OS working on devices that only have 128 MiB of RAM. The codename for these devices is Tarako (https://wiki.mozilla.org/FirefoxOS/Tarako). In case it's not obvious, the memory situation on these devices is *tight*. Optimizations that wouldn't have been worthwhile in the desktop-only days are now worthwhile. For example, an optimization that saves 100 KiB of memory per process is pretty worthwhile for Firefox OS. Thanks for the heads-up. Time to throw https://bugzilla.mozilla.org/show_bug.cgi?id=807359 back on the barbie... (Feel free to give it a new memshrink priority as you see fit, Nicholas) Jason ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
