Could we made those large strings or arrays copy-on-write? In OS level, we probably can make new pages COW[1]. Or we can implement it in SpiderMonkey[2]. That would not only save memory, but also improve performance I guess.
By the way, low memory warnings can be utilized better. Currently, on B2G, a flag in sysfs is checked every 5 seconds. This obviously is not the best. Maybe we could just make low memory killer send singal 35 before SIGKILL. Together with tunning the thresholds, exceptions should happen rarely. [1] I tried to mmap() /proc/$pid/mem but failed; /proc/$pid/mem can't be memory mapped. Some systems seem to have SHM_COPY to shmat(), but Linux seems not. Still trying to find a solution. [2] There should be some performance overheads. Not sure if most of the write-checks can be optimized away by JITs. ----- Original Message ----- From: "Justin Lebar" <[email protected]> To: "Thinker K.F. Li" <[email protected]> Cc: [email protected], [email protected], [email protected] Sent: Saturday, April 27, 2013 10:22:34 PM Subject: Re: [b2g] Bug 850175 > 3. tirgger scanning for low-memory warning. We have learned not to rely on low-memory warnings. We should still use them, but we should consider them to be an emergency measure which may or may not work. The problem is, often a program allocates too fast to see the low-memory warning. For example, in bug 865929, we have a cache of images that are drawn to canvases. That cache was becoming very large and causing us to crash, so we'd assumed (e.g. in the bug title) that the cache did not listen to memory pressure events. But it turns out, the cache /does/ listen to low-memory events, but we don't act on those events quickly enough to prevent a crash. I expect we can invoke KSM off the main thread, so we could run it sooner than we can run a GC, for example. But still, I don't think we should rely on it. The safest thing to do, I think, is not to copy the string many times. On Sat, Apr 27, 2013 at 2:37 AM, Thinker K.F. Li <[email protected]> wrote: > From: Ting-Yuan Huang <[email protected]> > Subject: Re: Bug 850175 > Date: Fri, 26 Apr 2013 21:09:50 -0700 (PDT) > >> KSM requires those strings to be aligned (to same offsets to page >> boundaries). It should be fine in this case, but I'm not quite sure. > > For jemalloc, it is quite sure for big memory allocation. For js > string, Greg told me we can use external string object. I think we > can make sure page alignment at external string object. > >> >> Another characteristic of KSM is that it scans periodically. From the >> discussion on bugzilla it seems that we are suffering from peak memory >> usage. I'm afraid that the original, unmodified KSM can't really help. I'll >> try to find if there are ways in userspace to make duplicated pages COW. > > I had looked into the code of KSM. If I am right, we can mark all big > strings after it was created, and trigger KSM to do scaning and merging > for low-memory warning. Then, these big string will be merged at the > time, low-memory warning. Another issue is the number of pages of > scanning is limited. We should pick a good one. > > With following recipe, I guess the big string will be merged in time. > 1. advise big strings after it is created and filled. > 2. perodically trigger scanning by write to /sys/kernel/mm/ksm/run. (opt) > 3. tirgger scanning for low-memory warning. > >> >> ----- Original Message ----- >> From: "Thinker K.F. Li" <[email protected]> >> To: [email protected] >> Cc: [email protected] >> Sent: Saturday, April 27, 2013 1:58:10 AM >> Subject: Re: Bug 850175 >> >> I had told to Greg. He told me the same string will be duplicated for >> 15 times for inserting to indexedDB. For indexedDb, it had duplicate >> it for at least 6 times. I think KSM can play a good game here. >> >> KSM can play good by advising only big strings or alikes, it play a >> trade-off of overhead and memory. We can trigger it to start scanning >> and merging for low memory. >> >> From: Thinker K.F. Li <[email protected]> >> Subject: Bug 850175 >> Date: Sat, 27 Apr 2013 00:20:22 +0800 (CST) >> >>> Hi Ting-Yuan, >>> >>> Tonight, people are talking about bug 850175 on #b2g channel. There >>> are two issues in that bug, one of issues is twitter will create a big >>> string and send it to indexedDB. It causes a lot of string >>> duplications in the peak. Since you are trying the kernel feature of >>> samepage merging, I guess it is a good solution to solve it. We can >>> give advisement only to big strings to reduce loading of scanning. >>> What do you think? >>> >>> see https://bugzilla.mozilla.org/show_bug.cgi?id=850175#c74 > _______________________________________________ > dev-b2g mailing list > [email protected] > https://lists.mozilla.org/listinfo/dev-b2g _______________________________________________ dev-b2g mailing list [email protected] https://lists.mozilla.org/listinfo/dev-b2g
