Re: Stability and Memory Pressure in 8.2

2008-10-07 Thread Tomeu Vizoso
On Wed, Oct 1, 2008 at 12:50 PM, S Page [EMAIL PROTECTED] wrote: Tomeu Vizoso wrote: Read has serious memory problems because renders whole pages into memory, regardless of what is the viewed area. Any chance the first pages of the PDF you opened contained big images? Nope, it's a saved

Re: Stability and Memory Pressure in 8.2

2008-10-07 Thread Sayamindu Dasgupta
On Tue, Sep 30, 2008 at 1:50 PM, Tomeu Vizoso [EMAIL PROTECTED] wrote: On Tue, Sep 30, 2008 at 4:13 AM, S Page [EMAIL PROTECTED] wrote: On September 8th, Michael Stone wrote: Kim, Greg, and I have concluded that the instability we experience under memory-pressure in 8.2-759 and similar is the

Re: Stability and Memory Pressure in 8.2

2008-10-01 Thread S Page
Tomeu Vizoso wrote: Read has serious memory problems because renders whole pages into memory, regardless of what is the viewed area. Any chance the first pages of the PDF you opened contained big images? Nope, it's a saved Project Gutenberg PDF

Re: Stability and Memory Pressure in 8.2

2008-09-30 Thread Tomeu Vizoso
On Tue, Sep 30, 2008 at 4:13 AM, S Page [EMAIL PROTECTED] wrote: On September 8th, Michael Stone wrote: Kim, Greg, and I have concluded that the instability we experience under memory-pressure in 8.2-759 and similar is the single hard issue that we wish to _attempt_ to address before releasing

Re: Stability and Memory Pressure in 8.2

2008-09-29 Thread S Page
On September 8th, Michael Stone wrote: Kim, Greg, and I have concluded that the instability we experience under memory-pressure in 8.2-759 and similar is the single hard issue that we wish to _attempt_ to address before releasing 8.2 on current timeframes. How did it go? I was going through

Re: Stability and Memory Pressure in 8.2

2008-09-14 Thread Marco Pesenti Gritti
On Sun, Sep 14, 2008 at 6:42 AM, James Cameron [EMAIL PROTECTED] wrote: I recall someone noticed that the animated activity icon was redrawing the whole screen. I think it got fixed. Since it got fixed, I haven't seen as many OOMs during olpc-update. It was not fixed.

Re: Stability and Memory Pressure in 8.2

2008-09-11 Thread C. Scott Ananian
On Wed, Sep 10, 2008 at 8:13 AM, Tomeu Vizoso [EMAIL PROTECTED] wrote: On Wed, Sep 10, 2008 at 2:05 PM, James Cameron [EMAIL PROTECTED] wrote: But I did notice one odd thing that I wasn't fully aware of until now ... the byte-code of the built-in modules was present, complete with doc strings

Re: Stability and Memory Pressure in 8.2

2008-09-11 Thread Tomeu Vizoso
On Thu, Sep 11, 2008 at 5:30 PM, C. Scott Ananian [EMAIL PROTECTED] wrote: On Wed, Sep 10, 2008 at 8:13 AM, Tomeu Vizoso [EMAIL PROTECTED] wrote: On Wed, Sep 10, 2008 at 2:05 PM, James Cameron [EMAIL PROTECTED] wrote: But I did notice one odd thing that I wasn't fully aware of until now ...

Re: Stability and Memory Pressure in 8.2

2008-09-11 Thread C. Scott Ananian
On Thu, Sep 11, 2008 at 12:16 PM, Tomeu Vizoso [EMAIL PROTECTED] wrote: On Thu, Sep 11, 2008 at 5:30 PM, C. Scott Ananian [EMAIL PROTECTED] wrote: On Wed, Sep 10, 2008 at 8:13 AM, Tomeu Vizoso [EMAIL PROTECTED] wrote: On Wed, Sep 10, 2008 at 2:05 PM, James Cameron [EMAIL PROTECTED] wrote: But

Re: Stability and Memory Pressure in 8.2

2008-09-11 Thread Tomeu Vizoso
On Thu, Sep 11, 2008 at 6:51 PM, C. Scott Ananian [EMAIL PROTECTED] wrote: On Thu, Sep 11, 2008 at 12:16 PM, Tomeu Vizoso [EMAIL PROTECTED] wrote: On Thu, Sep 11, 2008 at 5:30 PM, C. Scott Ananian [EMAIL PROTECTED] wrote: On Wed, Sep 10, 2008 at 8:13 AM, Tomeu Vizoso [EMAIL PROTECTED] wrote:

Re: Stability and Memory Pressure in 8.2

2008-09-10 Thread Marco Pesenti Gritti
On Wed, Sep 10, 2008 at 4:29 AM, Gary C Martin [EMAIL PROTECTED] wrote: OK, news is not great on the Activity front... SUMMARY: 759 vs 711 each Activity instance in 759 consumes an average of 1Mb more memory than the same Activity running in 711, with Write-57 reportedly taking significantly

Re: Stability and Memory Pressure in 8.2

2008-09-10 Thread Marco Pesenti Gritti
On Wed, Sep 10, 2008 at 4:29 AM, Gary C Martin [EMAIL PROTECTED] wrote: Well, I was hoping to see the numbers go the other way with the rainbow fork trick sharing more module code between Activities. Could be worse I guess – I should also test opening N instances of the same Activity and see

Re: Stability and Memory Pressure in 8.2

2008-09-10 Thread John Gilmore
When measuring memory usage, cat /proc/XXX/smaps provides the most accurate info available (as far as I know), and produces directly comparable results in all OLPC software releases. XXX is the process number you're examining (first column of ps output). The smaps file also tells you how many of

Re: Stability and Memory Pressure in 8.2

2008-09-10 Thread Tomeu Vizoso
On Wed, Sep 10, 2008 at 11:53 AM, John Gilmore [EMAIL PROTECTED] wrote: It may be possible and useful to store some commonly used executables and shared libraries as uncompressed files in jffs2, making them much faster to page back in from Flash. Nobody has tried doing this, as far as I

Re: Stability and Memory Pressure in 8.2

2008-09-10 Thread Tomeu Vizoso
On Tue, Sep 9, 2008 at 6:10 AM, Michael Stone [EMAIL PROTECTED] wrote: * We need to check carefully for memory-leaks. Three mechanisms which occur to me include: Looks like we have regressed on http://dev.laptop.org/ticket/5532 . Just entered http://dev.laptop.org/ticket/8394 because most

Re: Stability and Memory Pressure in 8.2

2008-09-10 Thread riccardo
On Wed, 2008-09-10 at 09:15 +0200, Marco Pesenti Gritti wrote: On Wed, Sep 10, 2008 at 4:29 AM, Gary C Martin [EMAIL PROTECTED] wrote: OK, news is not great on the Activity front... SUMMARY: 759 vs 711 each Activity instance in 759 consumes an average of 1Mb more memory than the same

Re: Stability and Memory Pressure in 8.2

2008-09-10 Thread James Cameron
I had a few hours look at the second largest process, the journal activity, on Joyride 2412. VmPeak:40440 kB VmSize:40436 kB VmLck: 0 kB VmHWM: 28824 kB VmRSS: 28824 kB VmData:11632 kB VmStk: 172 kB VmExe: 4 kB VmLib: 21992 kB VmPTE:48 kB so

Re: Stability and Memory Pressure in 8.2

2008-09-10 Thread pgf
tomeu wrote: On Wed, Sep 10, 2008 at 2:05 PM, James Cameron [EMAIL PROTECTED] wrote: Has anyone got an idea of how to measure the heap by usage? Not from outside python, but from inside we are using heapy: http://guppy-pe.sourceforge.net/ i started down that path yesterday

Re: Stability and Memory Pressure in 8.2

2008-09-10 Thread riccardo
Paul, On Wed, 2008-09-10 at 08:18 -0400, [EMAIL PROTECTED] wrote: tomeu wrote: On Wed, Sep 10, 2008 at 2:05 PM, James Cameron [EMAIL PROTECTED] wrote: Has anyone got an idea of how to measure the heap by usage? Not from outside python, but from inside we are using heapy:

Re: Stability and Memory Pressure in 8.2

2008-09-10 Thread Tomeu Vizoso
On Wed, Sep 10, 2008 at 2:37 PM, riccardo [EMAIL PROTECTED] wrote: Paul, On Wed, 2008-09-10 at 08:18 -0400, [EMAIL PROTECTED] wrote: tomeu wrote: On Wed, Sep 10, 2008 at 2:05 PM, James Cameron [EMAIL PROTECTED] wrote: Has anyone got an idea of how to measure the heap by usage?

Re: Stability and Memory Pressure in 8.2

2008-09-10 Thread david
On Wed, 10 Sep 2008, Tomeu Vizoso wrote: On Wed, Sep 10, 2008 at 11:53 AM, John Gilmore [EMAIL PROTECTED] wrote: It may be possible and useful to store some commonly used executables and shared libraries as uncompressed files in jffs2, making them much faster to page back in from Flash.

Re: Stability and Memory Pressure in 8.2

2008-09-10 Thread pgf
tomeu wrote: On Wed, Sep 10, 2008 at 2:37 PM, riccardo [EMAIL PROTECTED] wrote: Paul, On Wed, 2008-09-10 at 08:18 -0400, [EMAIL PROTECTED] wrote: i started down that path yesterday afternoon, and realized that it wasn't clear to me how i needed to invoke it. it seems to want

Re: Stability and Memory Pressure in 8.2

2008-09-10 Thread Michael Stone
On Wed, Sep 10, 2008 at 02:13:24PM +0200, Tomeu Vizoso wrote: Not from outside python, but from inside we are using heapy: http://guppy-pe.sourceforge.net/ Tomeu already published some guppy RPMs but here is a git repo with pacakging instructions (Makefiles) should you wish to make any changes.

Re: Stability and Memory Pressure in 8.2

2008-09-10 Thread Nate Ridderman
My layman's understanding is that you can't execute in place from the NAND flash on the XO. XIP requires NOR flash which is more expensive than NAND but has faster read speeds. It mentions this briefly on the axfs FAQ. Storing some executables and libraries in a separate uncompressed partition

Re: Stability and Memory Pressure in 8.2

2008-09-10 Thread Gary C Martin
On 10 Sep 2008, at 08:27, Marco Pesenti Gritti wrote: On Wed, Sep 10, 2008 at 4:29 AM, Gary C Martin [EMAIL PROTECTED] wrote: Well, I was hoping to see the numbers go the other way with the rainbow fork trick sharing more module code between Activities. Could be worse I guess – I should

Re: Stability and Memory Pressure in 8.2

2008-09-10 Thread Michael Stone
A more accurate test would be to disable the preloading itself rather than disabling isolation but leaving rainbow loading the libraries. :) To do that, see lines 31-32 of /usr/lib/python2.5/site_packages/rainbow/service.py You want to set self.preloader_hint = False and comment out the call

Re: Stability and Memory Pressure in 8.2

2008-09-09 Thread Tomeu Vizoso
On Tue, Sep 9, 2008 at 6:10 AM, Michael Stone [EMAIL PROTECTED] wrote: * We need to find out why the oom-killer is not killing things fast enough. Based on our results, we might consider configuring /proc/$pid/oom_adj to preferentially kill some processes (e.g., the foreground [or

Re: Stability and Memory Pressure in 8.2

2008-09-09 Thread Martin Dengler
On Tue, Sep 09, 2008 at 12:10:53AM -0400, Michael Stone wrote: Dear devel@, Kim, Greg, and I have concluded that the instability we experience under memory-pressure in 8.2-759 and similar is the single hard issue that we wish to _attempt_ to address before releasing 8.2 on current

Re: Stability and Memory Pressure in 8.2

2008-09-09 Thread Jim Gettys
There are four classes of things we can/should/could do: 1) understand where our memory is being used. Individual bugs can have a large effect. Something stupid could be hurting us badly, and we won't know unless we look. What is more, we need to invest in tools that allow us to monitor

Re: Stability and Memory Pressure in 8.2

2008-09-09 Thread riccardo
On Tue, 2008-09-09 at 00:10 -0400, Michael Stone wrote: Dear devel@, Kim, Greg, and I have concluded that the instability we experience under memory-pressure in 8.2-759 and similar is the single hard issue that we wish to _attempt_ to address before releasing 8.2 on current timeframes. (We

Re: Stability and Memory Pressure in 8.2

2008-09-09 Thread pgf
On Tue, 2008-09-09 at 00:10 -0400, Michael Stone wrote: - This means that we need to measure how our memory consumption profile has changed since our previous releases. (cscott observes that we were unable to attack the F-9 image size issues until we were able to

Re: Stability and Memory Pressure in 8.2 ([EMAIL PROTECTED])

2008-09-09 Thread Greg Smith
Hi All, I recommend build 708 as the baseline. Thanks, Greg S ** Date: Tue, 09 Sep 2008 11:34:50 -0400 From: [EMAIL PROTECTED] Subject: Re: Stability and Memory Pressure in 8.2 To: devel@lists.laptop.org Message-ID: [EMAIL PROTECTED] Content-Type: text/plain; charset=us

Re: Stability and Memory Pressure in 8.2

2008-09-09 Thread Deepak Saxena
* We need to find out why the oom-killer is not killing things fast enough. Based on our results, we might consider configuring /proc/$pid/oom_adj to preferentially kill some processes (e.g., the foreground [or background?] activities.) In the cases I've been playing with,

Re: Stability and Memory Pressure in 8.2

2008-09-09 Thread Daniel Drake
On Tue, 2008-09-09 at 00:10 -0400, Michael Stone wrote: - whether we can get Browse to behave intelligently when it receives BadAlloc errors from X? I have no doubt that Browse/xulrunner has room for improvement with memory usage but this is not where you should be looking. These

Re: Stability and Memory Pressure in 8.2

2008-09-09 Thread C. Scott Ananian
On Tue, Sep 9, 2008 at 11:34 AM, [EMAIL PROTECTED] wrote: On Tue, 2008-09-09 at 00:10 -0400, Michael Stone wrote: - This means that we need to measure how our memory consumption profile has changed since our previous releases. (cscott observes that we were unable to attack

Re: Stability and Memory Pressure in 8.2

2008-09-09 Thread C. Scott Ananian
On Tue, Sep 9, 2008 at 7:02 AM, Tomeu Vizoso [EMAIL PROTECTED] wrote: On Tue, Sep 9, 2008 at 6:10 AM, Michael Stone [EMAIL PROTECTED] wrote: * We need to find out why the oom-killer is not killing things fast enough. Based on our results, we might consider configuring

Re: Stability and Memory Pressure in 8.2

2008-09-09 Thread Tomeu Vizoso
On Tue, Sep 9, 2008 at 7:39 PM, C. Scott Ananian [EMAIL PROTECTED] wrote: On Tue, Sep 9, 2008 at 7:02 AM, Tomeu Vizoso [EMAIL PROTECTED] wrote: On Tue, Sep 9, 2008 at 6:10 AM, Michael Stone [EMAIL PROTECTED] wrote: * We need to find out why the oom-killer is not killing things fast

Re: Stability and Memory Pressure in 8.2

2008-09-09 Thread Jim Gettys
On Tue, 2008-09-09 at 13:10 -0400, Daniel Drake wrote: On Tue, 2008-09-09 at 00:10 -0400, Michael Stone wrote: - whether we can get Browse to behave intelligently when it receives BadAlloc errors from X? I have no doubt that Browse/xulrunner has room for improvement with memory

Re: Stability and Memory Pressure in 8.2

2008-09-09 Thread Michael Stone
On Tue, Sep 09, 2008 at 01:10:57PM -0400, Daniel Drake wrote: On Tue, 2008-09-09 at 00:10 -0400, Michael Stone wrote: - whether we can get Browse to behave intelligently when it receives BadAlloc errors from X? I have no doubt that Browse/xulrunner has room for improvement with memory

Re: Stability and Memory Pressure in 8.2

2008-09-09 Thread pgf
c. scott ananian wrote: On Tue, Sep 9, 2008 at 7:02 AM, Tomeu Vizoso [EMAIL PROTECTED] wrote: stability issue? AFAIK, we haven't seen OOM conditions without any activity open. Yes, we have. In particular, if you update your system and then leave it for a while, and later click the

Re: Stability and Memory Pressure in 8.2

2008-09-09 Thread pgf
i wrote: i think there's definitely a sugar shell leak. here's some partial data, gathered from a few machines on my desk right now. (be careful with the column headings -- i rearranged partway through to get separate CODE and DATA columns.) (also, don't do an absolute compare

Re: Stability and Memory Pressure in 8.2

2008-09-09 Thread Tomeu Vizoso
On Tue, Sep 9, 2008 at 9:13 PM, [EMAIL PROTECTED] wrote: i wrote: i think there's definitely a sugar shell leak. here's some partial data, gathered from a few machines on my desk right now. (be careful with the column headings -- i rearranged partway through to get separate

Re: Stability and Memory Pressure in 8.2

2008-09-09 Thread pgf
tomeu wrote: On Tue, Sep 9, 2008 at 9:13 PM, [EMAIL PROTECTED] wrote: (there are a lot of variables in play here -- the main thing is that something's certainly leaking.) The shell shouldn't be doing anything while idle, so checking if the trigger is activity network would help

Re: Stability and Memory Pressure in 8.2

2008-09-09 Thread Tomeu Vizoso
On Tue, Sep 9, 2008 at 9:24 PM, [EMAIL PROTECTED] wrote: tomeu wrote: On Tue, Sep 9, 2008 at 9:13 PM, [EMAIL PROTECTED] wrote: (there are a lot of variables in play here -- the main thing is that something's certainly leaking.) The shell shouldn't be doing anything while idle,

Re: Stability and Memory Pressure in 8.2

2008-09-09 Thread Martin Dengler
On Tue, Sep 09, 2008 at 03:13:28PM -0400, [EMAIL PROTECTED] wrote: [759 sugar shell leak] seems more like 4.5 MB/hour. joyride-2399 sitting back at home with no activities, doing nothing all day: -bash-3.2# uptime 18:14:19 up 20:46, 8 users, load average: 0.15, 0.09, 0.12 -bash-3.2#

Re: Stability and Memory Pressure in 8.2

2008-09-09 Thread Marco Pesenti Gritti
A couple of low risk fixes which could save ~6 mb at startup: Remove numpy usage from the shell http://dev.laptop.org/ticket/8372 (has patch) gst usage in the shell wastes 2.6mb http://dev.laptop.org/ticket/8375 Marco ___ Devel mailing list

Re: Stability and Memory Pressure in 8.2

2008-09-09 Thread C. Scott Ananian
On Tue, Sep 9, 2008 at 4:01 PM, Marco Pesenti Gritti [EMAIL PROTECTED] wrote: A couple of low risk fixes which could save ~6 mb at startup: Remove numpy usage from the shell http://dev.laptop.org/ticket/8372 (has patch) gst usage in the shell wastes 2.6mb http://dev.laptop.org/ticket/8375

Re: Stability and Memory Pressure in 8.2

2008-09-09 Thread Deepak Saxena
On Tue, Sep 09, 2008 at 05:10:41PM +, Deepak Saxena wrote: * We need to find out why the oom-killer is not killing things fast enough. Based on our results, we might consider configuring /proc/$pid/oom_adj to preferentially kill some processes (e.g., the foreground [or

Re: Stability and Memory Pressure in 8.2

2008-09-09 Thread Mikus Grinbergs
Remove numpy usage from the shell I have not been following this thread - but: There were several Activities (not just Measure) which used 'numeric'. Then 'numeric' was removed from the builds. I don't know what those Activities are using now. My concern is that if they happened to switch

Re: Stability and Memory Pressure in 8.2

2008-09-09 Thread Marco Pesenti Gritti
numpy will still work fine for activities. Marco On Tue, Sep 9, 2008 at 10:52 PM, Mikus Grinbergs [EMAIL PROTECTED] wrote: Remove numpy usage from the shell I have not been following this thread - but: There were several Activities (not just Measure) which used 'numeric'. Then 'numeric'

Re: Stability and Memory Pressure in 8.2

2008-09-09 Thread Gary C Martin
On 9 Sep 2008, at 05:10, Michael Stone wrote: * We need to determine why we encounter low-memory and out-of-memory situations more frequently than in previous releases. - This means that we need to measure how our memory consumption profile has changed since our previous releases.

Re: Stability and Memory Pressure in 8.2

2008-09-09 Thread Gary C Martin
On 10 Sep 2008, at 00:11, Gary C Martin wrote: SUMMARY: 759 vs 711 is only eating an extra ~16Mb of ram after a clean boot (no running Activities) **I'll try and test several Activity versions that can run on both builds and see how their individual resources have changed, will post later.

Stability and Memory Pressure in 8.2

2008-09-08 Thread Michael Stone
Dear devel@, Kim, Greg, and I have concluded that the instability we experience under memory-pressure in 8.2-759 and similar is the single hard issue that we wish to _attempt_ to address before releasing 8.2 on current timeframes. (We recognize that there are several other issues marked as