Re: Almost 50% less free memory in joyride-2302 compared with Update.1 (708)
On Sat, 2008-08-23 at 21:31 +1000, James Cameron wrote:
On Sat, Aug 23, 2008 at 12:01:07PM +0200, Marco Pesenti Gritti wrote:
On Fri, Aug 22, 2008 at 1:47 PM, James Cameron [EMAIL PROTECTED]
wrote:
...
Btw it's a shame that the python processes are grouped all together. I
wonder if we can fix ps_mem to show them separately, with the full
command.
I agree.
The attached patch makes ps_mem show the process cmdline instead of its
name; it doesn't account for the /proc/[pid]/cmdline particular format
but output it's well readable.
riccardo
--- ps_mem.orig 2008-07-22 08:00:58.0 +0200
+++ ps_mem 2008-08-25 10:01:35.0 +0200
@@ -117,14 +117,16 @@
return (Private, Shared)
def getCmdName(pid):
-cmd = file(/proc/%d/status % pid).readline()[6:-1]
-exe = os.path.basename(os.path.realpath(/proc/%d/exe % pid))
-if exe.startswith(cmd):
-cmd=exe #show non truncated version
-#Note because we show the non truncated name
-#one can have separated programs as follows:
-#584.0 KiB + 1.0 MiB = 1.6 MiBmozilla-thunder (exe - bash)
-# 56.0 MiB + 22.2 MiB = 78.2 MiBmozilla-thunderbird-bin
+cmd = file(/proc/%d/cmdline % pid).readline()[:60]
+if not len(cmd):
+cmd = file(/proc/%d/status % pid).readline()[6:-1]
+exe = os.path.basename(os.path.realpath(/proc/%d/exe % pid))
+if exe.startswith(cmd):
+cmd=exe #show non truncated version
+#Note because we show the non truncated name
+#one can have separated programs as follows:
+#584.0 KiB + 1.0 MiB = 1.6 MiBmozilla-thunder (exe - bash)
+# 56.0 MiB + 22.2 MiB = 78.2 MiBmozilla-thunderbird-bin
return cmd
cmds={}
___
Devel mailing list
Devel@lists.laptop.org
http://lists.laptop.org/listinfo/devel
Re: Cerebro (was Re: Almost 50% less free memory in joyride-2302 compared with Update.1 (708))
I did the following as a basic test for memory usage of various python modules. I did the test on my computer, a 32-bit Centrino laptop running ubuntu. Let's start with the following code: import gobject mainloop = gobject.MainLoop() mainloop.run() Cost: 1.2MB of memory (1.2MB total) Add a dummy class, like the following: import dbus import dbus.service class Dummy(dbus.service.Object): pass Additional cost: 0.9MB of memory (2.1MB total) Importing some more modules, like the following: import time, sys, os, traceback from select import select from socket import * from random import random from struct import pack, unpack from optparse import OptionParser from os.path import basename Additional cost: 1.0MB of memory (3.1MB total) Running the same thing as above as root: Additional cost: 2.8MB of memory (5.9MB total) (WHY?!?!?) Nothing specific to cerebro up to this point. Running Cerebro fully-blown as root: Additional cost: 1.3MB of memory (7.2MB total) It doesn't seem to me that there's much I can do, except for writing some parts of Cerebro in C. Pol -- Polychronis Ypodimatopoulos Graduate student Viral Communications MIT Media Lab Tel: +1 (617) 459-6058 http://www.mit.edu/~ypod/ ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Cerebro (was Re: Almost 50% less free memory in joyride-2302 compared with Update.1 (708))
On Sun, 24 Aug 2008, Polychronis Ypodimatopoulos wrote: I did the following as a basic test for memory usage of various python modules. I did the test on my computer, a 32-bit Centrino laptop running ubuntu. Let's start with the following code: import gobject mainloop = gobject.MainLoop() mainloop.run() Cost: 1.2MB of memory (1.2MB total) Add a dummy class, like the following: import dbus import dbus.service class Dummy(dbus.service.Object): pass Additional cost: 0.9MB of memory (2.1MB total) Importing some more modules, like the following: import time, sys, os, traceback from select import select from socket import * from random import random from struct import pack, unpack from optparse import OptionParser from os.path import basename Additional cost: 1.0MB of memory (3.1MB total) Running the same thing as above as root: Additional cost: 2.8MB of memory (5.9MB total) (WHY?!?!?) I wouldn't be surprised to leard that one of the modules included a bunch of extra functionality if you are root. for example dbus may find a bunch more things that you can do if you are root. it would probably be worthwhile narrowing down which module(s) increase their size if you are root. David Lang Nothing specific to cerebro up to this point. Running Cerebro fully-blown as root: Additional cost: 1.3MB of memory (7.2MB total) It doesn't seem to me that there's much I can do, except for writing some parts of Cerebro in C. Pol ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Cerebro (was Re: Almost 50% less free memory in joyride-2302 compared with Update.1 (708))
On Fri, Aug 22, 2008 at 1:47 PM, James Cameron [EMAIL PROTECTED] wrote: I also tried using ps_mem.py to compare memory used by processes, but the method that ps_mem.py uses has changed as a result of kernel change. http://dev.laptop.org/~quozl/2008-08-22-ps-mem/ has the samples that I took, showing how they are difficult to compare. Cerebro is running there for some reason (I've seen it in another bug report already). I can't reproduce on a clean 2328 joyride, not sure if it has been fixed or something weird is going on. Marco ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Almost 50% less free memory in joyride-2302 compared with Update.1 (708)
On Fri, Aug 22, 2008 at 1:47 PM, James Cameron [EMAIL PROTECTED] wrote: I've 703 and 2311 available to compare. Here's a few things I found. -- A test to compare available memory by eliminating buffer cache. Method: boot, wait for UI to be stable, switch to text console, echo 3 /proc/sys/vm/drop_caches, run free(1) and note the free column, repeat a few times. Result: build 703 has 123528 pages free, build 2311 has 93716 pages free. Which row are you looking at here? In -/+ buffers/cache I get 130368. This is joyride-2328. ps_mem gives me your same output, except for cerebro. Marco ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Almost 50% less free memory in joyride-2302 compared with Update.1 (708)
On Fri, Aug 22, 2008 at 1:47 PM, James Cameron [EMAIL PROTECTED] wrote: I also tried using ps_mem.py to compare memory used by processes, but the method that ps_mem.py uses has changed as a result of kernel change. http://dev.laptop.org/~quozl/2008-08-22-ps-mem/ has the samples that I took, showing how they are difficult to compare. How they are difficult to compare? Btw it's a shame that the python processes are grouped all together. I wonder if we can fix ps_mem to show them separately, with the full command. Marco ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Almost 50% less free memory in joyride-2302 compared with Update.1 (708)
On Sat, Aug 23, 2008 at 11:36:17AM +0200, Marco Pesenti Gritti wrote: Which row are you looking at here? In -/+ buffers/cache I get 130368. This is joyride-2328. That test was using the free value on the first line, which matches MemFree in /proc/meminfo. I chose that because I knew that after the cache drop the only cache remaining was needed by the processes that were active. Looking at -/+ buffers/cache for this test would not have been useful, since the cached pages are effectively unreleasable despite any memory demand. I did look at the values, but there was no point reporting them. The value you report seems normal. ps_mem gives me your same output, except for cerebro. I've no idea why cerebro chose to run. Under what conditions is it run? -- James Cameronmailto:[EMAIL PROTECTED] http://quozl.netrek.org/ ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Almost 50% less free memory in joyride-2302 compared with Update.1 (708)
On Sat, Aug 23, 2008 at 12:01:07PM +0200, Marco Pesenti Gritti wrote: On Fri, Aug 22, 2008 at 1:47 PM, James Cameron [EMAIL PROTECTED] wrote: I also tried using ps_mem.py to compare memory used by processes, but the method that ps_mem.py uses has changed as a result of kernel change. http://dev.laptop.org/~quozl/2008-08-22-ps-mem/ has the samples that I took, showing how they are difficult to compare. How they are difficult to compare? They are difficult to compare *because* of the different method by which ps_mem accounts for shared memory. Every process was changed in some way, but when you realise that the cause included a different accounting method, you see how difficult they are to compare. Take for instance the cat process, which should not have changed unduly from 703 to 2311. In the ps_mem of 703 the RAM used estimate is 464 KiB, but in 2311 it is 100 KiB. While it looks great, it doesn't match other evidence. The change is mainly due to the different accounting for shared memory. The private memory changed from 72 KiB in build 703 to 76 KiB in build 2311. Btw it's a shame that the python processes are grouped all together. I wonder if we can fix ps_mem to show them separately, with the full command. I agree. -- James Cameronmailto:[EMAIL PROTECTED] http://quozl.netrek.org/ ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Cerebro (was Re: Almost 50% less free memory in joyride-2302 compared with Update.1 (708))
On Sat, Aug 23, 2008 at 5:16 AM, Marco Pesenti Gritti [EMAIL PROTECTED] wrote: Cerebro is running there for some reason (I've seen it in another bug report already). I can't reproduce on a clean 2328 joyride, not sure if it has been fixed or something weird is going on. Cerebro is enabled in joyride builds, started from /etc/init.d; it should be disabled in 8.2 builds at the moment. How much memory usage is cerebro responsible for? --scott -- ( http://cscott.net/ ) ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Cerebro (was Re: Almost 50% less free memory in joyride-2302 compared with Update.1 (708))
On Sat, Aug 23, 2008 at 3:09 PM, C. Scott Ananian [EMAIL PROTECTED] wrote: On Sat, Aug 23, 2008 at 5:16 AM, Marco Pesenti Gritti [EMAIL PROTECTED] wrote: Cerebro is running there for some reason (I've seen it in another bug report already). I can't reproduce on a clean 2328 joyride, not sure if it has been fixed or something weird is going on. Cerebro is enabled in joyride builds, started from /etc/init.d; it should be disabled in 8.2 builds at the moment. How much memory usage is cerebro responsible for? 6.5 Mib according to ps_mem You are right, it's enabled in joyride... it's failing for me because it can't find msh0 (not sure why). Marco ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Cerebro (was Re: Almost 50% less free memory in joyride-2302 compared with Update.1 (708))
On Sat, Aug 23, 2008 at 1:25 PM, Marco Pesenti Gritti [EMAIL PROTECTED] wrote: On Sat, Aug 23, 2008 at 3:09 PM, C. Scott Ananian [EMAIL PROTECTED] wrote: On Sat, Aug 23, 2008 at 5:16 AM, Marco Pesenti Gritti [EMAIL PROTECTED] wrote: Cerebro is running there for some reason (I've seen it in another bug report already). I can't reproduce on a clean 2328 joyride, not sure if it has been fixed or something weird is going on. Cerebro is enabled in joyride builds, started from /etc/init.d; it should be disabled in 8.2 builds at the moment. How much memory usage is cerebro responsible for? 6.5 Mib according to ps_mem You are right, it's enabled in joyride... it's failing for me because it can't find msh0 (not sure why). Marco ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel Same for me. Joyride 2325 has cerebro enabled, but it aborts because device msh0 cannot be found. This also hangs up the shutdown when trying to kill cerebro. So somehow cerebro is enabled in 2325. Also idle suspend is enabled again in 2325, while it should be off. The msh0 disappearing might have something to do with the new libertas firmware. I also have much more problems with wireless connections. Will do some more testing. ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Cerebro (was Re: Almost 50% less free memory in joyride-2302 compared with Update.1 (708))
ton van overbeek wrote: Also idle suspend is enabled again in 2325, while it should be off. i think this was intentional. now that we're building separate release (Subject: New release8.2 build 7nn) and joyride streams, idle suspend was re-enabled in joyride in order to help continue flushing out suspend-related bugs. paul =- paul fox, [EMAIL PROTECTED] ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Cerebro (was Re: Almost 50% less free memory in joyride-2302 compared with Update.1 (708))
Marco Pesenti Gritti wrote: 6.5 Mib according to ps_mem You are right, it's enabled in joyride... it's failing for me because it can't find msh0 (not sure why). Marco I filed #8128 to address the memory usage that seems excessive. I have also disabled cerebro from start-up while this is being investigated and the issue with blocking shutdown process (#8108). Should be picked up at the next version of joyride. p. ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Almost 50% less free memory in joyride-2302 compared with Update.1 (708)
On Thu, 21 Aug 2008, Ton van Overbeek wrote: Noticed the following: After booting with all the joyride activities installed and then running top on vt2 (no activities running, only journal) joyride-2302 shows 44268k free. Booting into 708 with the same set of activities shows 82828k free. Are you taking the the buffers/cache space into account? -- Asheesh. -- From Sharp minds come... pointed heads. -- Bryan Sparrowhawk ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Almost 50% less free memory in joyride-2302 compared with Update.1 (708)
On Fri, Aug 22, 2008 at 1:52 PM, Ton van Overbeek [EMAIL PROTECTED] wrote: Noticed the following: After booting with all the joyride activities installed and then running top on vt2 (no activities running, only journal) joyride-2302 shows 44268k free. Booting into 708 with the same set of activities shows 82828k free. Is there a 'min amount of free memory' release criterion/goal for 8.2 ??? It seems OOM situations will occur more often in 8.2 than in 8.1. Can you use one of the nice smap-summary-listing utilities and prepare a before-and-after (or a diff if that's workable)? That'll help narrow down on what's hogging your memory. Reading % memory free from top or vmstat is not particularly informative, specially in this day and age when the kernel and modern code do all sorts of COW-type tricks... This one seems to be one of the nicest ones: http://www.pixelbeat.org/scripts/ps_mem.py cheers, m -- [EMAIL PROTECTED] [EMAIL PROTECTED] -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: Almost 50% less free memory in joyride-2302 compared with Update.1 (708)
I've 703 and 2311 available to compare. Here's a few things I found.
--
A test to compare available memory by eliminating buffer cache.
Method: boot, wait for UI to be stable, switch to text console, echo 3
/proc/sys/vm/drop_caches, run free(1) and note the free column, repeat
a few times.
Result: build 703 has 123528 pages free, build 2311 has 93716 pages
free.
Conclusion: the extra functions in build 2311 over build 703 consume an
extra 30Mb of memory, before any user interaction.
--
A test to consume free memory with cache to encourage page out.
Method: with UI stable, in text console use find /usr -type f -exec cp
{} /tmp/j \; to consume free memory, in a second text console monitor
free(1) until free memory has reached a minimum, then in the original
text console hit Ctrl/C, echo 3 /proc/sys/vm/drop_caches, run free(1)
and note the free column.
Result: no significant change in free values from previous test.
Conclusion: that without swap backed process pages are probably not
removed from memory.
--
I also tried using ps_mem.py to compare memory used by processes, but
the method that ps_mem.py uses has changed as a result of kernel change.
http://dev.laptop.org/~quozl/2008-08-22-ps-mem/ has the samples that I
took, showing how they are difficult to compare.
--
James Cameronmailto:[EMAIL PROTECTED] http://quozl.netrek.org/
___
Devel mailing list
Devel@lists.laptop.org
http://lists.laptop.org/listinfo/devel
Almost 50% less free memory in joyride-2302 compared with Update.1 (708)
Noticed the following: After booting with all the joyride activities installed and then running top on vt2 (no activities running, only journal) joyride-2302 shows 44268k free. Booting into 708 with the same set of activities shows 82828k free. Is there a 'min amount of free memory' release criterion/goal for 8.2 ??? It seems OOM situations will occur more often in 8.2 than in 8.1. Ton van Overbeek ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
