Re: ASU - out of memory?

[The Rasterman]

On Fri, 22 Aug 2008 11:13:19 +0200 Tilman Baumann [EMAIL PROTECTED] babbled:

 Pardon.
 I don't care for the warm and fuzzy feeling you get by having malloc  
 fail on you.

the problem is.. it doesn't say it failed. it says it succeeded. it returns a
pointer. program now moves on and should be able to safely assume that the
memory it asked for is it's to use. overcommit makes success from malloc a
gamble. it may or may not provide you with usable memory. this makes any kind
of attempt to do error checking pretty useless. as success can also be an
error that is not detectable (until it's too late and you've segv'd).
 
 It does not give you a bit more system stability! The one app  
 receiving malloc errors is just not app of many. They all have a  
 problem then.
 Imagine, the browser catches a failed malloc, because some other  
 stupid app has eaten almost all ram.
 What is the benefit of telling the browser about low mem? It could  
 only safe itself from crashing. Well done.

and that is what it SHOULD do. programs SHOULD check the return of their malloc
()'s and unwind safely and remain in a safe non-crashing state. they may
inform the user of such an issue or may chose some other method to deal with
it, but it is the job of every app to deal with it as gracefully as possible.

 And malloc btw. never gives you bad memory. You just have to check,  
 even with overcommit.

with overcommit it CAN give you bad memory. without it can't (or shouldn't - if
it did - it'd be a bug). NULL = no memory could be given to you of the size
you asked for. deal with it, anything else is giving you the memory you asked
for.

with overcommit it SAYS it gives you that memory - but once you try and access
it (read or write) at ANY time, you may or may not segv. because the kernel may
have overcommitted and decided to throw out that memory because it thinks it is
not used (for whatever reason), but now you want to use it.

let me give an example:

my app starts - i alloc 64kb for a quick lookup-table for a dictionary. (this
is a real-life thing i actually do btw.) this lets me find words in a
dictionary file when you type to match quickly (as opposed to do a full linear
search of the file). this lets me jump to a subsection (block) of the file
quickly based on the first few letters of a word then do a limited linear
search. this is low on memory usage as its a limited lookup table. this gets
set up on app start. now. lets say you don't type anything for a while - hours
or days. in the meantime you run apps (dialler, address lookup etc.) and just
passively browse the web (no entering fields - just using bookmarks). lots of
memory is allocated and freed, but the pages of ram for the keyboard dictionary
are untouched. it hasn't needed them. now you use your browser and it teats up
gobs of ram. well what happens now you need to type. you bring up the keyboard
and type in your name in an entry field. as it does this it may need to lookup a
dictionary. the kernel has since overcommitted and thrown away some pages in
the 64kb lookup table for the dictionary. it decided that since they werre idle
- they may aswell be ditched as it REALLY needed that ram for that background
jpeg in the web page! this was new, urgent and active. well - now as you type
the keyboard code tries to do a lookup.. and finds the pages of its dictionary
gone! kernel took them away. they were successfully allocated and filled, but
now have been nuked. result? keyboard segv's. no bug in kbd. it did everything
right. it gets punished for the memory hogging of another app. THAT is what
overcommit does.

 It's just so that you have to be aware that the kernel who is watching  
 over all resources (the evil MCP muahahah)
 may need to lay off some programms in a critical system condition.  
 (oom_kill)
 No one wants this, so the strategy is to avoid this. We are talking  
 here about the same thing. We need some userspace helper to clean up  
 before it gets critical.

yes. agreed. and we need to not use overcommit. you can't go punish apps that
have done everything right and by the book. overcommit does this.

 Yes, it sounds bad to have a error handling for out of memory and then  
 never use it.
 Ask my mon, she will tell you this is horrible. But my mom does not  
 know the whole picture.

its not just the error handling -the SUCCESS case is also now broken. see
above. you are liable to be able to segv if you handle the error case or not.

 And once again. Not having overcommit wastes memory. Why do you think  
 this is worth it?

and having it makes apps segv that would not have segv'd before. it wastes
developers chasing gdb backtraces and bug reports for something that doesn't
even exist as a bug. see above. you'd get an utterly bizarre crash somewhere it
just doesn't make sense - you'd possibly spend days or weeks hunting it to no
avail as it just isn't what you think it is! the kernel literally has removed
your ram from you! there is no bug to hunt!.

 

Re: ASU - out of memory?

[The Rasterman]

On Fri, 22 Aug 2008 12:36:52 -0400 Chris Wright [EMAIL PROTECTED] babbled:

 On the other hand, let's say your process allocates some memory and
 doesn't use it for a while. In the meantime, some memory is freed.
 This doesn't help if malloc() returned null, but it does help if the
 kernel overcommitted memory instead.
 
 I don't think that's as useful. But you could instead define a malloc:
 void* _malloc(size_t length)
 {
void* pointer = malloc(length);
if (!mlock(pointer, length)) return null; // or abort
return pointer;
 }

unfortunately... this isn't practical. only root can mlock(). sure. on om we
run all as root - but that is something that probably should change and is not
a sane solution. :)

-- 
Carsten Haitzler (The Rasterman) [EMAIL PROTECTED]

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: ASU - out of memory?

[The Rasterman]

On Fri, 22 Aug 2008 16:53:15 +0200 Clemens Kirchgatterer [EMAIL PROTECTED]
babbled:

 Carsten Haitzler (The Rasterman) [EMAIL PROTECTED] wrote:
 
   Problem is, your might not have the memory to trawl those .desktop
   files.
  
  thus my original write a root daemon - setsched() to realtime
  priority and mlock() memory space down! (and of course read every
  page of memory to make sure it's paged in) :)
 
 from man mlock:
 
 mlock() locks pages in the address range starting at addr and
 continuing for len  bytes. All  pages that contain a part of the
 specified address range are guaranteed to be resi‐ dent in RAM when the
 call returns successfully; the pages are guaranteed to stay in  RAM
 until later unlocked.
 
 so reading every page seams unnessesary to me.

aaah yes. that saves the read :)

-- 
Carsten Haitzler (The Rasterman) [EMAIL PROTECTED]

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: ASU - out of memory?

[Sander van Grieken]

On Friday 22 August 2008 02:59:25 Carsten Haitzler wrote:
 On Thu, 21 Aug 2008 19:54:26 +0200 Sander van Grieken [EMAIL PROTECTED] 
babbled:
  For a phone, the algorithm could be as simple as killing the process that
  has allocated the most memory. The essential system services and the
  basic UI applications usually have a small footprint, and the biggest
  consumer of memory is most likely a leaky UI app that's not part of the
  main system anyway.
 
  For a production server with large databases this doesn't work of course,
  but there you're already in big trouble if you have to fallback on the
  oom-killer.

 true - and the kernel oom killer should mostly handle this, BUT it is
 possible to do better. a userspace oom killer can trawl all .desktop files
 and thus know if the app is run by a user (base on command), or if its a
 system app that can be run, or if its installed later etc. etc.

 such a userspace oom isn't hard to do. it's pretty simple.

Problem is, your might not have the memory to trawl those .desktop files.

What about a malloc that's LD_PRELOADed in front of non-essential apps, that 
enforces a 5-10% available memory for essential system and phone apps? (or 
maybe some other hook mechanism if preloading is not feasible)

Of course there also should be such wrappers for delegated allocators like the 
X pixmap example your mentioned earlier.


Sander

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: ASU - out of memory?

[The Rasterman]

On Fri, 22 Aug 2008 08:36:26 +0200 Sander van Grieken [EMAIL PROTECTED] 
babbled:

 On Friday 22 August 2008 02:59:25 Carsten Haitzler wrote:
  On Thu, 21 Aug 2008 19:54:26 +0200 Sander van Grieken [EMAIL PROTECTED] 
 babbled:
   For a phone, the algorithm could be as simple as killing the process that
   has allocated the most memory. The essential system services and the
   basic UI applications usually have a small footprint, and the biggest
   consumer of memory is most likely a leaky UI app that's not part of the
   main system anyway.
  
   For a production server with large databases this doesn't work of course,
   but there you're already in big trouble if you have to fallback on the
   oom-killer.
 
  true - and the kernel oom killer should mostly handle this, BUT it is
  possible to do better. a userspace oom killer can trawl all .desktop files
  and thus know if the app is run by a user (base on command), or if its a
  system app that can be run, or if its installed later etc. etc.
 
  such a userspace oom isn't hard to do. it's pretty simple.
 
 Problem is, your might not have the memory to trawl those .desktop files.

thus my original write a root daemon - setsched() to realtime priority and
mlock() memory space down! (and of course read every page of memory to make
sure it's paged in) :)

 What about a malloc that's LD_PRELOADed in front of non-essential apps, that 
 enforces a 5-10% available memory for essential system and phone apps? (or 
 maybe some other hook mechanism if preloading is not feasible)
 
 Of course there also should be such wrappers for delegated allocators like
 the X pixmap example your mentioned earlier.

yeah. though such a userspace oom could use xresource extension to track x
client usage too and do the same as it would rummaging through /proc
regularly :)

 Sander
 
 ___
 Openmoko community mailing list
 community@lists.openmoko.org
 http://lists.openmoko.org/mailman/listinfo/community


-- 
Carsten Haitzler (The Rasterman) [EMAIL PROTECTED]

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: ASU - out of memory?

[Tilman Baumann]

Am 22.08.2008 um 02:50 schrieb Carsten Haitzler (The Rasterman):


 A 128Mio space of memory is not that huge, especially if you want to
 use it for something cool and you will at some points pass this mark
 and things will die; given the current algorithm it might not be
 something you want to get killed.

 128m is massive. it's unimaginably huge. if you need more you need  
 to look hard
 at what it is you are doing, and how. seriously. for the kind of  
 processing
 power, storage ability and display ability of this device... - and  
 no. not
 saying 640k is enough for everyone. i am saying that 128m is a  
 lot. people
 who don't think so are incredibly spoilt. possibly are bad  
 programmers who need
 to learn more about memory management or should have their  
 programming licenses
 revoked! :) (ok just kidding there). but seriously - it's a lot.  
 this is why
 instead of running off to swap for help - which just overflows your  
 ram problems
 onto disk (and sooner or later swap fills - then what? you haven't  
 fixed the
 original problem - just delayed it).


Yes, 128mb is much.
Yes, memory problems don't go away with swap. They are just a bit more  
easy to handle.
Yes, Overflowing swap is still a problem. But this means no one cared  
to solve the problem first.

And even if 128mb are much. Why waste memory?
(overcommit off + emergency threshold  100% for save app shutdown)

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: ASU - out of memory?

[Tilman Baumann]

Pardon.
I don't care for the warm and fuzzy feeling you get by having malloc  
fail on you.

It does not give you a bit more system stability! The one app  
receiving malloc errors is just not app of many. They all have a  
problem then.
Imagine, the browser catches a failed malloc, because some other  
stupid app has eaten almost all ram.
What is the benefit of telling the browser about low mem? It could  
only safe itself from crashing. Well done.

And malloc btw. never gives you bad memory. You just have to check,  
even with overcommit.
It's just so that you have to be aware that the kernel who is watching  
over all resources (the evil MCP muahahah)
may need to lay off some programms in a critical system condition.  
(oom_kill)
No one wants this, so the strategy is to avoid this. We are talking  
here about the same thing. We need some userspace helper to clean up  
before it gets critical.

Yes, it sounds bad to have a error handling for out of memory and then  
never use it.
Ask my mon, she will tell you this is horrible. But my mom does not  
know the whole picture.

And once again. Not having overcommit wastes memory. Why do you think  
this is worth it?

Ah, and btw. Nice analogy.

If you malloc (money alloc) some money at your bank they will give you  
some paper.
You know it is virtual value. You know they vastly overcommit.
You know, they will give you virtual money even if they can't know if  
they can keep that promise?

Sounds horrible, doesn't it?
It kind of is.  But they have swap too. And they try to fix it when  
they become aware that there is a problem.
If you are lucky the banking kernel finds a solution. If not, we are  
all screwed.

Come on, no one would ever do this! If they could not trust the paper  
they get, how could they know if there is a problem?
It's a horrible system, let's go back to real values like gold...

No, we don't. See you at the bank. ;)

Am 22.08.2008 um 02:33 schrieb Carsten Haitzler (The Rasterman):

 On Thu, 21 Aug 2008 18:40:52 +0200 Tilman Baumann  
 [EMAIL PROTECTED] babbled:


 Not being able to malloc memory and not having any physical memory  
 left
 are just two sepereate things. At least on modern linux systems.

 no they are not. i write code.

 myptr = malloc(mysize);

 it returns failuer (NULL) then i need to deal with it.
 it returns a pointer - it succeeded. i asked for memory - it gave it  
 to me. the
 problem is with overcommit success returns are lies. they MAY have  
 the memory -
 they may not. part way through using the pages it returned and SAID  
 i could
 have, it can just segv - as it is overcommitted and out of pages.  
 this means
 that suddenly return values can't be trusted for memory allocations  
 anymore.
 any attempts to handle NULL returns may as well not exist as success  
 cases can
 be undetectable failures. it's just a stupid policy. sure - its  
 there to work
 around stupid userspace code that goes off allocing massive blobs of  
 ram that
 it then never goes and uses, but the kernel should go punish all  
 apps for this
 - those apps being stupid should be punished and have their code  
 fixed.

 Memory overcommit saves (physical) memory. And not just a bit.
 And with some swap safety net it is reasonably save to do so.

 And how should it be better if _any_ app gets knocked out by  
 running in
 the memory limit? Usually it can't help the situation. Especially if
 some other app that is eating all memory.
 The effect is more or less the same. Some random poor app has to be
 killed. (Since suicide is often the only way to react to malloc  
 fails)

 Turning overcommit off is in my eyes only a poor 'bury one's head  
 in the
 sand' solution which effectively does improve nothing.

 i disagree. overcommit is a bury head in sand solution. it means  
 you just go
 and avoid the original problem of allocing much more memory than you  
 really
 need.

 Overcommit and swap is just the winner. Everyone does it so. And in  
 my
 eyes rightly so. It is fast and efficient.
 And even without swap, i would not want to turn overcommit off.

 Swap and overcommit is just the dream team. Fast and efficient  
 memory.
 And if something goes wrong, you have plenty of time (and mem) for
 solving the problem.
 Contrary to without, because how could you fix any low memory  
 condition
 with not allocating any more memory? Driving something into swap to  
 be
 able to do something about it is just right.

 And just if you thought so. Overcommit is not just lazy don't care  
 for
 errors behaviour but a really smart optimisation.

 wrong. it means i can no longer trust a successful malloc() calloc()  
 realloc()
 or even alloca() return. ever. the return of a valid pointer which  
 according to
 the manuals provided ever since for these calls, could be an error  
 state too.
 and thew only way to handle it is have your own sigsegv/bus handlers  
 and on a
 segv which accesses a known allocated chunk of memory, realise that  
 overcommit
 

Re: ASU - out of memory?

[Tilman Baumann]

Am 22.08.2008 um 02:50 schrieb Carsten Haitzler (The Rasterman):

 On Thu, 21 Aug 2008 18:49:33 +0200 Tilman Baumann  
 [EMAIL PROTECTED] babbled:

 And come on. Software is not perfect. Sometimes we have to live  
 with a
 dreamteam like (old) firefox and x11. I had times when they had both

 man it gets annoying people blaming x11 for memory problems. it is  
 rarely a
 cause. it LOOKs bad because it is allocating ram FOR client apps  
 (like firefox)
 - when firefox asks for 300mb of pixmaps.. the memory sits in x11 -  
 not in
 firefox.

Yea i know. This is how it should be. My X did not always free this  
crap.


 hundreds of megs virtual mem. But everything was fine because it  
 all was
 just harmlessly been swaped away. I restarted them every weekend to  
 not
 let it become worse.
 Not ideal, but should the system rather be unusable in this  
 condition?

 we should have a userspace oom that on low-memory kills off user  
 apps (not
 xserver or wm or basic required processes - just optional ones like  
 calculator,
 addressbook, web browser etc.) when things get low -
ACK
 not going and adding swap.
NACK


___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: ASU - out of memory?

[Clemens Kirchgatterer]

Carsten Haitzler (The Rasterman) [EMAIL PROTECTED] wrote:

  Problem is, your might not have the memory to trawl those .desktop
  files.
 
 thus my original write a root daemon - setsched() to realtime
 priority and mlock() memory space down! (and of course read every
 page of memory to make sure it's paged in) :)

from man mlock:

mlock() locks pages in the address range starting at addr and
continuing for len  bytes. All  pages that contain a part of the
specified address range are guaranteed to be resi‐ dent in RAM when the
call returns successfully; the pages are guaranteed to stay in  RAM
until later unlocked.

so reading every page seams unnessesary to me.

clemens

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: ASU - out of memory?

[Clemens Kirchgatterer]

Tilman Baumann [EMAIL PROTECTED] wrote:

 44kbyte are not very impressive, but they prove my point never the
 less. This memory is obviously waste. It stays in swap for ages. Not
 much saved, but anyhow. It is memory that did not need to be in ram.

Tilman, i agree to most points you made in this thread, but i think the
discussion is moot anyway. adding swap to our FR is easy enought, so
everyone can do what he thinks is right. a few years ago i also was one
of the people who thought not having a swap partition would make my
linux desktop faster. oh, dear. :-P

best regards ...
clemens

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: ASU - out of memory?

[Tilman Baumann]

Clemens Kirchgatterer wrote:
 Tilman Baumann [EMAIL PROTECTED] wrote:
 
 44kbyte are not very impressive, but they prove my point never the
 less. This memory is obviously waste. It stays in swap for ages. Not
 much saved, but anyhow. It is memory that did not need to be in ram.
 
 Tilman, i agree to most points you made in this thread, but i think the
 discussion is moot anyway. adding swap to our FR is easy enought, so
 everyone can do what he thinks is right. 
Which brings me to something i wanted to ask you.
Where is the no-overcommit switched on? I could not find it via grep in etc.
Is it a compile time default from the kernel? Or is there some setup 
script i overlooked?

Some automatic swapon if swapfile or partition is found on SD might be 
my first contribution to openmoko... :)

-- 
Drucken Sie diese Mail bitte nur auf Recyclingpapier aus.
Please print this mail only on recycled paper.

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: ASU - out of memory?

[Chris Wright]

2008/8/22 Tilman Baumann [EMAIL PROTECTED]:
 Pardon.
 I don't care for the warm and fuzzy feeling you get by having malloc
 fail on you.

 It does not give you a bit more system stability! The one app
 receiving malloc errors is just not app of many. They all have a
 problem then.
 Imagine, the browser catches a failed malloc, because some other
 stupid app has eaten almost all ram.
 What is the benefit of telling the browser about low mem? It could
 only safe itself from crashing. Well done.

Imagine that malloc() returns null rather than overcommitting.
Most affected apps will receive a SIGSEGV when malloc returns null
because there's no error handling for that situation.
Some will notice that malloc() returned null and either forgo some
optional stuff or abort a single operation or simply just shut down
gracefully.

Imagine that malloc() returns an invalid pointer when it overcommits.
All affected apps will receive a SIGSEGV in this case, without some
unusual error checking.

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: ASU - out of memory?

[Chris Wright]

2008/8/22 Chris Wright [EMAIL PROTECTED]:
 2008/8/22 Tilman Baumann [EMAIL PROTECTED]:
 Pardon.
 I don't care for the warm and fuzzy feeling you get by having malloc
 fail on you.

 It does not give you a bit more system stability! The one app
 receiving malloc errors is just not app of many. They all have a
 problem then.
 Imagine, the browser catches a failed malloc, because some other
 stupid app has eaten almost all ram.
 What is the benefit of telling the browser about low mem? It could
 only safe itself from crashing. Well done.

 Imagine that malloc() returns null rather than overcommitting.
 Most affected apps will receive a SIGSEGV when malloc returns null
 because there's no error handling for that situation.
 Some will notice that malloc() returned null and either forgo some
 optional stuff or abort a single operation or simply just shut down
 gracefully.

On the other hand, let's say your process allocates some memory and
doesn't use it for a while. In the meantime, some memory is freed.
This doesn't help if malloc() returned null, but it does help if the
kernel overcommitted memory instead.

I don't think that's as useful. But you could instead define a malloc:
void* _malloc(size_t length)
{
   void* pointer = malloc(length);
   if (!mlock(pointer, length)) return null; // or abort
   return pointer;
}

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: ASU - out of memory?

[Tilman Baumann]

Clemens Kirchgatterer wrote:
 Tilman Baumann [EMAIL PROTECTED] wrote:
 
 44kbyte are not very impressive, but they prove my point never the
 less. This memory is obviously waste. It stays in swap for ages. Not
 much saved, but anyhow. It is memory that did not need to be in ram.
 
 Tilman, i agree to most points you made in this thread, but i think the
 discussion is moot anyway. adding swap to our FR is easy enought, so
 everyone can do what he thinks is right.

Fair enough. :)
I made some tests see my other recent mail.
Let's see what the community makes out of it. I think i have started 
something important here. *g*

-- 
Drucken Sie diese Mail bitte nur auf Recyclingpapier aus.
Please print this mail only on recycled paper.

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: ASU - out of memory?

[Tilman Baumann]

Looks like we need swap after all.
Swap on flash sounds crazy, but this would be a place where non live 
pages can be dumped. I assume the majority of the memory that causes 
problems like this is leaked memory or regular bloat.

There seem to be some proposals how to make oom-kill behave more 
cooperative. But when oom-kill comes, every thing is too late in my eyes.

Torfinn Ingolfsen wrote:
 I am running ASU on my FreeRunner. After it being up for a day or two,
 things  (like the touch screen) stops working. ight now it has been up
 for:
 [EMAIL PROTECTED]:~# uptime
  22:53:34 up 3 days,  5:10,  2 users,  load average: 1.01, 1.15, 1.28
 [EMAIL PROTECTED]:~#
 
 From logread:
 Aug 20 22:11:47 om-gta02 user.warn kernel: [275268.785000] hald
 invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
 Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.785000]
 [c002dbfc] (dump_stack+0x0/0x14) from [c007a554]
 (oom_kill_process+0x58/0xec)
 Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.795000]
 [c007a4fc] (oom_kill_process+0x0/0xec) from [c007aa28]
 (out_of_memory+0x1a4/0x1fc)
 Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.805000]
 r7:0134 r6:c6884720 r5:c0387158 r4:c77e73c0
 Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.81]
 [c007a884] (out_of_memory+0x0/0x1fc) from [c007cdcc]
 (__alloc_pages+0x27c/0x308)
 Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.82]
 [c007cb50] (__alloc_pages+0x0/0x308) from [c007f068]
 (__do_page_cache_readahead+0x148/0x2b0)
 Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.83]
 [c007ef20] (__do_page_cache_readahead+0x0/0x2b0) from [c007f68c]
 (do_page_cache_readahead+0x70/0x80)
 Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.84]
 [c007f61c] (do_page_cache_readahead+0x0/0x80) from [c00799c0]
 (filemap_fault+0x1d4/0x454)
 Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.85]
 r7:c76f6620 r6:c68d2000 r5: r4:
 Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.855000]
 [c00797ec] (filemap_fault+0x0/0x454) from [c0084e04]
 (__do_fault+0x74/0x43c)
 Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.865000]
 [c0084d90] (__do_fault+0x0/0x43c) from [c0085f64]
 (handle_mm_fault+0x308/0x700)
 Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.87]
 [c0085c5c] (handle_mm_fault+0x0/0x700) from [c002fde0]
 (do_page_fault+0x100/0x23c)
 Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.88]
 [c002fce0] (do_page_fault+0x0/0x23c) from [c002ffd0]
 (do_translation_fault+0x20/0x80)
 Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.89]
 [c002ffb0] (do_translation_fault+0x0/0x80) from [c00281bc]
 (do_PrefetchAbort+0x18/0x1c)
 Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.90]
 r5:be9248c0 r4:
 Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.905000]
 [c00281a4] (do_PrefetchAbort+0x0/0x1c) from [c0028e80]
 (ret_from_exception+0x0/0x10)
 Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.915000] Exception
 stack(0xc68d3fb0 to 0xc68d3ff8)
 Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.92] 3fa0:
   00043660 000389f0 be9247b8 be9248b8
 Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.925000] 3fc0:
 be9248d4 be9248c0 be9248c4 be9248c8 be9248cc be9248d0 000412e0
 0001
 Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.935000] 3fe0:
 000414c4 be924778 0002c748 4024c050 2010 
 Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.94] Mem-info:
 Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.945000] DMA per-cpu:
 Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.945000] CPU0:
 Hot: hi:   42, btch:   7 usd:  40   Cold: hi:   14, btch:   3 usd:   9
 Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.95]
 Active:27461 inactive:310 dirty:0 writeback:0 unstable:0
 Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.95]  free:360
 slab:1883 mapped:4 pagetables:220 bounce:0
 Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.955000] DMA
 free:1440kB min:1440kB low:1800kB high:2160kB active:109844kB
 inactive:1240kB present:130048kB pages_scanned:193944
 all_unreclaimable? yes
 Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.96]
 lowmem_reserve[]: 0 0 0
 Aug 20 22:13:23 om-gta02 user.warn kernel: [275268.965000] DMA: 0*4kB
 2*8kB 1*16kB 0*32kB 0*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB
 0*4096kB = 1440kB
 Aug 20 22:13:23 om-gta02 user.warn kernel: [275268.975000] Swap cache:
 add 0, delete 0, find 0/0, race 0+0
 Aug 20 22:13:23 om-gta02 user.warn kernel: [275268.98] Free swap  = 0kB
 Aug 20 22:13:23 om-gta02 user.warn kernel: [275268.985000] Total swap = 0kB
 Aug 20 22:13:23 om-gta02 user.warn kernel: [275268.985000] Free swap:
   0kB
 Aug 20 22:13:23 om-gta02 user.warn kernel: [275269.00] 32768 pages of RAM
 Aug 20 22:13:23 om-gta02 user.warn kernel: [275269.00] 604 free pages
 Aug 20 22:13:23 om-gta02 user.warn kernel: [275269.00] 1924 reserved pages
 Aug 20 22:13:23 om-gta02 user.warn kernel: [275269.005000] 1883 

Re: ASU - out of memory?

[The Rasterman]

On Thu, 21 Aug 2008 11:42:52 +0200 Tilman Baumann [EMAIL PROTECTED] babbled:

no.. you need to find who is leaking memory and beat them up! :) seriously.
128m is more than enough. it's almost overkill. needing swap (on a device like
the freerunner) is a sign of stupid programming :)

 Looks like we need swap after all.
 Swap on flash sounds crazy, but this would be a place where non live 
 pages can be dumped. I assume the majority of the memory that causes 
 problems like this is leaked memory or regular bloat.
 
 There seem to be some proposals how to make oom-kill behave more 
 cooperative. But when oom-kill comes, every thing is too late in my eyes.
 
 Torfinn Ingolfsen wrote:
  I am running ASU on my FreeRunner. After it being up for a day or two,
  things  (like the touch screen) stops working. ight now it has been up
  for:
  [EMAIL PROTECTED]:~# uptime
   22:53:34 up 3 days,  5:10,  2 users,  load average: 1.01, 1.15, 1.28
  [EMAIL PROTECTED]:~#
  
  From logread:
  Aug 20 22:11:47 om-gta02 user.warn kernel: [275268.785000] hald
  invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.785000]
  [c002dbfc] (dump_stack+0x0/0x14) from [c007a554]
  (oom_kill_process+0x58/0xec)
  Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.795000]
  [c007a4fc] (oom_kill_process+0x0/0xec) from [c007aa28]
  (out_of_memory+0x1a4/0x1fc)
  Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.805000]
  r7:0134 r6:c6884720 r5:c0387158 r4:c77e73c0
  Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.81]
  [c007a884] (out_of_memory+0x0/0x1fc) from [c007cdcc]
  (__alloc_pages+0x27c/0x308)
  Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.82]
  [c007cb50] (__alloc_pages+0x0/0x308) from [c007f068]
  (__do_page_cache_readahead+0x148/0x2b0)
  Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.83]
  [c007ef20] (__do_page_cache_readahead+0x0/0x2b0) from [c007f68c]
  (do_page_cache_readahead+0x70/0x80)
  Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.84]
  [c007f61c] (do_page_cache_readahead+0x0/0x80) from [c00799c0]
  (filemap_fault+0x1d4/0x454)
  Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.85]
  r7:c76f6620 r6:c68d2000 r5: r4:
  Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.855000]
  [c00797ec] (filemap_fault+0x0/0x454) from [c0084e04]
  (__do_fault+0x74/0x43c)
  Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.865000]
  [c0084d90] (__do_fault+0x0/0x43c) from [c0085f64]
  (handle_mm_fault+0x308/0x700)
  Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.87]
  [c0085c5c] (handle_mm_fault+0x0/0x700) from [c002fde0]
  (do_page_fault+0x100/0x23c)
  Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.88]
  [c002fce0] (do_page_fault+0x0/0x23c) from [c002ffd0]
  (do_translation_fault+0x20/0x80)
  Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.89]
  [c002ffb0] (do_translation_fault+0x0/0x80) from [c00281bc]
  (do_PrefetchAbort+0x18/0x1c)
  Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.90]
  r5:be9248c0 r4:
  Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.905000]
  [c00281a4] (do_PrefetchAbort+0x0/0x1c) from [c0028e80]
  (ret_from_exception+0x0/0x10)
  Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.915000] Exception
  stack(0xc68d3fb0 to 0xc68d3ff8)
  Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.92] 3fa0:
00043660 000389f0 be9247b8 be9248b8
  Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.925000] 3fc0:
  be9248d4 be9248c0 be9248c4 be9248c8 be9248cc be9248d0 000412e0
  0001
  Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.935000] 3fe0:
  000414c4 be924778 0002c748 4024c050 2010 
  Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.94] Mem-info:
  Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.945000] DMA per-cpu:
  Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.945000] CPU0:
  Hot: hi:   42, btch:   7 usd:  40   Cold: hi:   14, btch:   3 usd:   9
  Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.95]
  Active:27461 inactive:310 dirty:0 writeback:0 unstable:0
  Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.95]  free:360
  slab:1883 mapped:4 pagetables:220 bounce:0
  Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.955000] DMA
  free:1440kB min:1440kB low:1800kB high:2160kB active:109844kB
  inactive:1240kB present:130048kB pages_scanned:193944
  all_unreclaimable? yes
  Aug 20 22:13:22 om-gta02 user.warn kernel: [275268.96]
  lowmem_reserve[]: 0 0 0
  Aug 20 22:13:23 om-gta02 user.warn kernel: [275268.965000] DMA: 0*4kB
  2*8kB 1*16kB 0*32kB 0*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB
  0*4096kB = 1440kB
  Aug 20 22:13:23 om-gta02 user.warn kernel: [275268.975000] Swap cache:
  add 0, delete 0, find 0/0, race 0+0
  Aug 20 22:13:23 om-gta02 user.warn kernel: [275268.98] Free swap  = 0kB
  Aug 20 22:13:23 om-gta02 user.warn kernel: [275268.985000] Total swap = 0kB
  Aug 20 

Re: ASU - out of memory?

[Tilman Baumann]

Carsten Haitzler (The Rasterman) wrote:
 On Thu, 21 Aug 2008 11:42:52 +0200 Tilman Baumann [EMAIL PROTECTED] babbled:
 
 no.. you need to find who is leaking memory and beat them up! :) seriously.
 128m is more than enough. it's almost overkill. needing swap (on a device like
 the freerunner) is a sign of stupid programming :)

I understand your Argument and it is true. But the conclusion is wrong.
Even if you don't have obvious leaks, some memory will not be used most 
of the time and can be safely dumped out of the RAM.
Like some background process which is sleeping almost all the time. Or 
other multitasked apps which are not used currently.
Or all the fat frameworks which probably take some memory for code or 
data which are almost never used.

All this memory is wasted on valuable RAM. Even if only a hand full of 
pages end on swap, they are saved from clogging up RAM.

And using huge amounts of virtual memory is not necessarily a effect of 
a memory leak. Virtual memory is almost free after all on all other 
linux systems.

And another point is just the more graceful effect when memory really 
becomes low.
You feel the speed impact. You feel that something needs to be done. Or 
  some popup can ask the user for some intervention - all before it is 
too late.

As i said, swap on flash sound crazy. But i had this experience on my 
Nokia 770 (64Mb RAM). Even if i was not always on the memory limit, it 
felt instantaneously better and more stable.

-- 
Drucken Sie diese Mail bitte nur auf Recyclingpapier aus.
Please print this mail only on recycled paper.

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: ASU - out of memory?

[Rui Miguel Silva Seabra]

On Thu, Aug 21, 2008 at 12:58:44PM +0200, Tilman Baumann wrote:
 Carsten Haitzler (The Rasterman) wrote:
  On Thu, 21 Aug 2008 11:42:52 +0200 Tilman Baumann [EMAIL PROTECTED] 
  babbled:
  
  no.. you need to find who is leaking memory and beat them up! :) seriously.
  128m is more than enough. it's almost overkill. needing swap (on a device 
  like
  the freerunner) is a sign of stupid programming :)
 
 I understand your Argument and it is true. But the conclusion is wrong.
 Even if you don't have obvious leaks, some memory will not be used most 
 of the time and can be safely dumped out of the RAM.
 Like some background process which is sleeping almost all the time. Or 
 other multitasked apps which are not used currently.
 Or all the fat frameworks which probably take some memory for code or 
 data which are almost never used.
 
 All this memory is wasted on valuable RAM. Even if only a hand full of 
 pages end on swap, they are saved from clogging up RAM.

But every access to swap will mean a significant decrease of battery
life.

Rui

-- 
Keep the Lasagna flying!
Today is Pungenday, the 14th day of Bureaucracy in the YOLD 3174
+ No matter how much you do, you never do enough -- unknown
+ Whatever you do will be insignificant,
| but it is very important that you do it -- Gandhi
+ So let's do it...?

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: ASU - out of memory?

[Tilman Baumann]

Rui Miguel Silva Seabra wrote:
 On Thu, Aug 21, 2008 at 12:58:44PM +0200, Tilman Baumann wrote:
 Carsten Haitzler (The Rasterman) wrote:
 On Thu, 21 Aug 2008 11:42:52 +0200 Tilman Baumann [EMAIL PROTECTED] 
 babbled:

 no.. you need to find who is leaking memory and beat them up! :) seriously.
 128m is more than enough. it's almost overkill. needing swap (on a device 
 like
 the freerunner) is a sign of stupid programming :)
 I understand your Argument and it is true. But the conclusion is wrong.
 Even if you don't have obvious leaks, some memory will not be used most 
 of the time and can be safely dumped out of the RAM.
 Like some background process which is sleeping almost all the time. Or 
 other multitasked apps which are not used currently.
 Or all the fat frameworks which probably take some memory for code or 
 data which are almost never used.

 All this memory is wasted on valuable RAM. Even if only a hand full of 
 pages end on swap, they are saved from clogging up RAM.
 
 But every access to swap will mean a significant decrease of battery
 life.

There are probably some kernel-knobs to fiddle with to make the kernel 
behave very conservative in regards to how early it pages something out.
A ideal swap would only be filled with stuff that is not to be read 
again for ages.
Or in other words, it should be as un-dynamic as possible.


-- 
Drucken Sie diese Mail bitte nur auf Recyclingpapier aus.
Please print this mail only on recycled paper.

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: ASU - out of memory?

[The Rasterman]

On Thu, 21 Aug 2008 12:58:44 +0200 Tilman Baumann [EMAIL PROTECTED] babbled:

 Carsten Haitzler (The Rasterman) wrote:
  On Thu, 21 Aug 2008 11:42:52 +0200 Tilman Baumann [EMAIL PROTECTED]
  babbled:
  
  no.. you need to find who is leaking memory and beat them up! :) seriously.
  128m is more than enough. it's almost overkill. needing swap (on a device
  like the freerunner) is a sign of stupid programming :)
 
 I understand your Argument and it is true. But the conclusion is wrong.
 Even if you don't have obvious leaks, some memory will not be used most 
 of the time and can be safely dumped out of the RAM.
 Like some background process which is sleeping almost all the time. Or 
 other multitasked apps which are not used currently.
 Or all the fat frameworks which probably take some memory for code or 
 data which are almost never used.

given the abysmal IO speed flah and sdio can manage - not to mention the
overhead and waitstates we get from swapping via glamo... i'd sooner want to
tear out my eyeballs than have anything swap over SD...

 All this memory is wasted on valuable RAM. Even if only a hand full of 
 pages end on swap, they are saved from clogging up RAM.
 
 And using huge amounts of virtual memory is not necessarily a effect of 
 a memory leak. Virtual memory is almost free after all on all other 
 linux systems.
 
 And another point is just the more graceful effect when memory really 
 becomes low.
 You feel the speed impact. You feel that something needs to be done. Or 
   some popup can ask the user for some intervention - all before it is 
 too late.
 
 As i said, swap on flash sound crazy. But i had this experience on my 
 Nokia 770 (64Mb RAM). Even if i was not always on the memory limit, it 
 felt instantaneously better and more stable.

i think its just the lazy mans way out. there's a problem .. look - lets
cover it up by swapping!. 128m ram is a huge amount. most of the time a vast
chunk of it (at least 25%) be cache - i.e. memory you dont need - but it just
kept around in case you need it again soon. the only thing we are talking here
is using swap in favor of using that space for disk cache really. given we have
things like sd card corruption problems on suspend/resume... not to mention
performance issues, i'd think it would make little sense so make the system
that fragile. fix the memory usage issue - don't just get more slow slow slow
ram. :)

-- 
Carsten Haitzler (The Rasterman) [EMAIL PROTECTED]

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: ASU - out of memory?

[Esben Stien]

Tilman Baumann [EMAIL PROTECTED] writes:

 all the linux memory overcommit behaviour more or less depends on
 the fact that it can allways save it's ass by using swap. (Instead
 of helplessley crashing)

Yes, or killing the application. Not having swap is nonsense;). If you
are using swap something is wrong, right, but then you fix it. I find
it strange that the debian install didn't make a little swap
partition.

-- 
Esben Stien is [EMAIL PROTECTED] s  a 
 http://www. s tn m
  irc://irc.  b  -  i  .   e/%23contact
   sip:b0ef@   e e 
   jid:b0ef@n n

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: ASU - out of memory?

[Tilman Baumann]

Tilman Baumann wrote:

 i'd think it would make little sense so make the system
 that fragile. fix the memory usage issue - don't just get more slow slow 
 slow
 ram. :)
 
 As i said, i had made the experience that it improved the system 
 greatly. Shouldn't we just test it?
 Perhaps, that is what i will try to do with my Neo1973 soon...

This is how swap was used after launching some apps simultaniously 
(tangogps, navit, browser, mediaplayer) using them for some time and 
then shut them down again:
  total   used   free sharedbuffers cached
Mem:126268 111008  15260  0   1304  68736
-/+ buffers/cache:  40968  85300
Swap:64720 44  64676

These 44kbytes came only some minutes after swap was enabled and used. 
And they came only in a few chunks.

44kbyte are not very impressive, but they prove my point never the less.
This memory is obviously waste. It stays in swap for ages. Not much 
saved, but anyhow. It is memory that did not need to be in ram.

I would assume with more complex software like it is coming for 
freerunners (enough ROM for bloatware) this will sightly increase.

And regarding the point of wasting power with io on the swapfile.
Sure, i did not measure real io. But just looking at the output of free 
  via watch shows pretty clearly that swap is virtually not used in 
regular usage.
Ergo, in case no emergency is present, swap is not used and no harm is 
done. (Besides saving some kbyte of real RAM)
But IF memory becomes a problem, swap comes to rescue.

(Found no real world way to really excessively use mem and drive the 
kernel into swapping in the short time. But the result seems pretty 
obvious to me. No crash but graceful performance breakdown)

Based on this quick survey, i would say there no way swap could bring 
any danger or adverse effects.
-- 
Drucken Sie diese Mail bitte nur auf Recyclingpapier aus.
Please print this mail only on recycled paper.

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: ASU - out of memory?

[Tilman Baumann]

Esben Stien wrote:
 Tilman Baumann [EMAIL PROTECTED] writes:
 
 all the linux memory overcommit behaviour more or less depends on
 the fact that it can allways save it's ass by using swap. (Instead
 of helplessley crashing)
 
 Yes, or killing the application. 

That would be great. But was always a hot topic and AFAIK no solution 
was found yet.
I remember there was a proposal for a more intelligent oom killer system 
on lkml some time ago. No idea if this is still around.
(Was afaik some preemtive notification to userspace)

Intuitively the answer is clear. Kill the culprit and not just something 
big. But what is to kill? And how to determine without allocating any 
more memory...
I have never seen any craceful oom killer yet.

-- 
Drucken Sie diese Mail bitte nur auf Recyclingpapier aus.
Please print this mail only on recycled paper.

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: ASU - out of memory?

[The Rasterman]

On Thu, 21 Aug 2008 17:50:26 +0200 Esben Stien [EMAIL PROTECTED] babbled:

 Tilman Baumann [EMAIL PROTECTED] writes:
 
  all the linux memory overcommit behaviour more or less depends on
  the fact that it can allways save it's ass by using swap. (Instead
  of helplessley crashing)
 
 Yes, or killing the application. Not having swap is nonsense;). If you
 are using swap something is wrong, right, but then you fix it. I find
 it strange that the debian install didn't make a little swap
 partition.

and luckily those smart fellas in kernel developer land.. made kernel
overcommit.. a tunable parameter! and... cunningly.. on the FR (and as wel on
my desktop) it's turned off! :) so... a moot point really. :)

-- 
Carsten Haitzler (The Rasterman) [EMAIL PROTECTED]

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: ASU - out of memory?

[The Rasterman]

On Thu, 21 Aug 2008 16:09:58 +0200 Tilman Baumann [EMAIL PROTECTED] babbled:

 Esben Stien wrote:
  Tilman Baumann [EMAIL PROTECTED] writes:
  
  all the linux memory overcommit behaviour more or less depends on
  the fact that it can allways save it's ass by using swap. (Instead
  of helplessley crashing)
  
  Yes, or killing the application. 
 
 That would be great. But was always a hot topic and AFAIK no solution 
 was found yet.
 I remember there was a proposal for a more intelligent oom killer system 
 on lkml some time ago. No idea if this is still around.
 (Was afaik some preemtive notification to userspace)
 
 Intuitively the answer is clear. Kill the culprit and not just something 
 big. But what is to kill? And how to determine without allocating any 
 more memory...
 I have never seen any craceful oom killer yet.

really - we need a userspace oom that is Smarter (it knows what a system
daemon is and what a user application is and what is a necessary user desktop
process), so it will always kill apps not the phone daemon or the window
manager or the launcher etc. it will stick to finding visible apps that have
gone rogue and nuke them. the userspace oom can run as root - mlock() its
memoryspace in and give itself a realtime priority with setsched(). every
second or so - check the memory state of the system, if it begins to get low -
issue a warning signal to processes (could use dbus or something more basic
like system signals - use SIGURG or something pretty much unused these days).
any well behaved app when it gets such a signal should:

* free any memory it doesnt REALLY need (any caches or any data loaded that it
can get back from a file again), so save out stuff and free.
* if it does garbage collection - collect now. asap!
* if it makes sense - exit the process altogether (if it is really not a
useful process and it can be started later when things get better)

this would - allow things to clean up nicely and avoid swap storms. remember
swap comes with the cost of writing to swap AND reading back at another time.
throwing out pages from disk cache and then just loading them later if u need
them again has less cost - throwing out costs nothing, and paging in is (give
or take) the same cost as from swap. the kernel makes a wild guess as to what
pages are really so unused they should be swapped as cache is more useful than
those pages... but better than making the kernel HAVE to throw them into swap
is.. get rid of them entirely!

yes.. this requires work on the part of app authors in userspace. but it's the
right thing to do (imho).

anyway - the same userspace oom daemon can then choose to kill processes should
the situation become dire - and in this case it could be sure NOT to kill X,
window manager, launcher, dbus daemon etc. (important system processes that
keep the ui and user-facing bits that keep the thing usable).

:)

-- 
Carsten Haitzler (The Rasterman) [EMAIL PROTECTED]

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: ASU - out of memory?

[Tilman Baumann]

Carsten Haitzler (The Rasterman) wrote:
 On Thu, 21 Aug 2008 17:50:26 +0200 Esben Stien [EMAIL PROTECTED] babbled:
 
 Tilman Baumann [EMAIL PROTECTED] writes:

 all the linux memory overcommit behaviour more or less depends on
 the fact that it can allways save it's ass by using swap. (Instead
 of helplessley crashing)
 Yes, or killing the application. Not having swap is nonsense;). If you
 are using swap something is wrong, right, but then you fix it. I find
 it strange that the debian install didn't make a little swap
 partition.
 
 and luckily those smart fellas in kernel developer land.. made kernel
 overcommit.. a tunable parameter! and... cunningly.. on the FR (and as wel on
 my desktop) it's turned off! :) so... a moot point really. :)

pardon? Honestly? This is absurd!
Why? I don't get it.

I mean, how did you get the impression that overcomitting is a bad thing?

-- 
Drucken Sie diese Mail bitte nur auf Recyclingpapier aus.
Please print this mail only on recycled paper.

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: ASU - out of memory?

[Tilman Baumann]

Carsten Haitzler (The Rasterman) wrote:

 I remember there was a proposal for a more intelligent oom killer system 
 on lkml some time ago. No idea if this is still around.
 (Was afaik some preemtive notification to userspace)

 Intuitively the answer is clear. Kill the culprit and not just something 
 big. But what is to kill? And how to determine without allocating any 
 more memory...
 I have never seen any craceful oom killer yet.
 
 really - we need a userspace oom that is Smarter (it knows what a system
 daemon is and what a user application is and what is a necessary user desktop
 process), so it will always kill apps not the phone daemon or the window
 manager or the launcher etc. it will stick to finding visible apps that have
 gone rogue and nuke them.

http://kerneltrap.org/node/850
Had not followed the story, but seems to go in the right direction


-- 
Drucken Sie diese Mail bitte nur auf Recyclingpapier aus.
Please print this mail only on recycled paper.

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: ASU - out of memory?

[Tilman Baumann]

Tilman Baumann wrote:
 Carsten Haitzler (The Rasterman) wrote:
 
 I remember there was a proposal for a more intelligent oom killer system 
 on lkml some time ago. No idea if this is still around.
 (Was afaik some preemtive notification to userspace)

 Intuitively the answer is clear. Kill the culprit and not just something 
 big. But what is to kill? And how to determine without allocating any 
 more memory...
 I have never seen any craceful oom killer yet.
 really - we need a userspace oom that is Smarter (it knows what a system
 daemon is and what a user application is and what is a necessary user 
 desktop
 process), so it will always kill apps not the phone daemon or the window
 manager or the launcher etc. it will stick to finding visible apps that 
 have
 gone rogue and nuke them.
 
 http://kerneltrap.org/node/850
 Had not followed the story, but seems to go in the right direction

As i said. Never ending story.
I was not sure this was the story i remembered. So i searched a bit 
further...
http://kerneltrap.org/Linux/Out_of_Memory_Notification
http://kerneltrap.org/Linux/Signaling_When_Out_of_Memory

As you can see, they are well aware of the problem. But somehow there is 
no universal solution yet. (of which i know)

But it seems to me like we could use someone to dig into this topic and 
find a good solution. (I'm not really a VM guy)
Because clearly, plain oom-kill is bad. However the vm handles memory.

Maybe a stupid solution like looking for unusual growth of swap usage 
might even do the trick.
But i really like the SIGDANGER idea...
-- 
Drucken Sie diese Mail bitte nur auf Recyclingpapier aus.
Please print this mail only on recycled paper.

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: ASU - out of memory?

[The Rasterman]

On Thu, 21 Aug 2008 16:44:36 +0200 Tilman Baumann [EMAIL PROTECTED] babbled:

 Carsten Haitzler (The Rasterman) wrote:
  On Thu, 21 Aug 2008 17:50:26 +0200 Esben Stien [EMAIL PROTECTED]
  babbled:
  
  Tilman Baumann [EMAIL PROTECTED] writes:
 
  all the linux memory overcommit behaviour more or less depends on
  the fact that it can allways save it's ass by using swap. (Instead
  of helplessley crashing)
  Yes, or killing the application. Not having swap is nonsense;). If you
  are using swap something is wrong, right, but then you fix it. I find
  it strange that the debian install didn't make a little swap
  partition.
  
  and luckily those smart fellas in kernel developer land.. made kernel
  overcommit.. a tunable parameter! and... cunningly.. on the FR (and as wel
  on my desktop) it's turned off! :) so... a moot point really. :)
 
 pardon? Honestly? This is absurd!
 Why? I don't get it.
 
 I mean, how did you get the impression that overcomitting is a bad thing?

when it makes:


myptr = malloc(somesize);
if (myptr == NULL) return -1; // return and unwind
// continue as normal


useless so i mayaswell do:

myptr = malloc(somesize);
if (myptr == NULL) abort();


ala glib. all that error checking code programs have becomes stupid. i should
just assume all mallocs succeed!. as they likely will and part way through
accessing the memory malloc actually returned - i may segv because i dont have
enough ram. that's just bad! what's the point of error codes and returns then?
whats the point of even trying to handle an error? :( may as well throw in the
towel ala glib.

-- 
Carsten Haitzler (The Rasterman) [EMAIL PROTECTED]

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: ASU - out of memory?

[Sander van Grieken]

 Tilman Baumann [EMAIL PROTECTED] writes:

 all the linux memory overcommit behaviour more or less depends on
 the fact that it can allways save it's ass by using swap. (Instead
 of helplessley crashing)

 Yes, or killing the application. Not having swap is nonsense;). If you
 are using swap something is wrong, right, but then you fix it. I find
 it strange that the debian install didn't make a little swap
 partition.

How is having 256MB RAM without swap any different from 128MB RAM with 128MB 
swap? It's
still 256MB (virtual) memeory in total.

In other words, even WITH swap you dont solve the problem, you only postpone any
problems until later, AND you lose some flash/sd storage room for swap 
permanently.

It only makes sense to allocate swap if you know beforehand you'll need more 
than the
available RAM.

Sander



___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: ASU - out of memory?

[Tilman Baumann]

Carsten Haitzler (The Rasterman) wrote:
 On Thu, 21 Aug 2008 16:44:36 +0200 Tilman Baumann [EMAIL PROTECTED] babbled:
 
 Carsten Haitzler (The Rasterman) wrote:
 On Thu, 21 Aug 2008 17:50:26 +0200 Esben Stien [EMAIL PROTECTED]
 babbled:

 Tilman Baumann [EMAIL PROTECTED] writes:

 all the linux memory overcommit behaviour more or less depends on
 the fact that it can allways save it's ass by using swap. (Instead
 of helplessley crashing)
 Yes, or killing the application. Not having swap is nonsense;). If you
 are using swap something is wrong, right, but then you fix it. I find
 it strange that the debian install didn't make a little swap
 partition.
 and luckily those smart fellas in kernel developer land.. made kernel
 overcommit.. a tunable parameter! and... cunningly.. on the FR (and as wel
 on my desktop) it's turned off! :) so... a moot point really. :)
 pardon? Honestly? This is absurd!
 Why? I don't get it.

 I mean, how did you get the impression that overcomitting is a bad thing?
 
 when it makes:
 
 
 myptr = malloc(somesize);
 if (myptr == NULL) return -1; // return and unwind
 // continue as normal
 
 
 useless so i mayaswell do:
 
 myptr = malloc(somesize);
 if (myptr == NULL) abort();
 
 
 ala glib. all that error checking code programs have becomes stupid. i should
 just assume all mallocs succeed!. as they likely will and part way through
 accessing the memory malloc actually returned - i may segv because i dont have
 enough ram. that's just bad! what's the point of error codes and returns then?
 whats the point of even trying to handle an error? :( may as well throw in the
 towel ala glib.
 

Not being able to malloc memory and not having any physical memory left 
are just two sepereate things. At least on modern linux systems.

Memory overcommit saves (physical) memory. And not just a bit.
And with some swap safety net it is reasonably save to do so.

And how should it be better if _any_ app gets knocked out by running in 
the memory limit? Usually it can't help the situation. Especially if 
some other app that is eating all memory.
The effect is more or less the same. Some random poor app has to be 
killed. (Since suicide is often the only way to react to malloc fails)

Turning overcommit off is in my eyes only a poor 'bury one's head in the 
sand' solution which effectively does improve nothing.

Overcommit and swap is just the winner. Everyone does it so. And in my 
eyes rightly so. It is fast and efficient.
And even without swap, i would not want to turn overcommit off.

Swap and overcommit is just the dream team. Fast and efficient memory. 
And if something goes wrong, you have plenty of time (and mem) for 
solving the problem.
Contrary to without, because how could you fix any low memory condition 
with not allocating any more memory? Driving something into swap to be 
able to do something about it is just right.

And just if you thought so. Overcommit is not just lazy don't care for 
errors behaviour but a really smart optimisation.
For example it saves memory by only copying a page if the page was 
written. (copy on write)
For example:
If you fork, the memory space is duplicated. You end up with twice the 
memory. But linux does not copy the memory, it overcommits.
Only if a page is really changed it get duplicated first.

This is elegant, fast and efficient. But yes, you loose the context of a 
error condition if some happens. But i say, if a system runs out of 
memory, letting a program know that there is no memory left helps 
nothing. The poor program that gehts the error is usually neither able 
to save much or even to solve the system wide problem.

-- 
Drucken Sie diese Mail bitte nur auf Recyclingpapier aus.
Please print this mail only on recycled paper.

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: ASU - out of memory?

[Tilman Baumann]

Sander van Grieken wrote:
 Tilman Baumann [EMAIL PROTECTED] writes:

 all the linux memory overcommit behaviour more or less depends on
 the fact that it can allways save it's ass by using swap. (Instead
 of helplessley crashing)
 Yes, or killing the application. Not having swap is nonsense;). If you
 are using swap something is wrong, right, but then you fix it. I find
 it strange that the debian install didn't make a little swap
 partition.
 
 How is having 256MB RAM without swap any different from 128MB RAM with 128MB 
 swap? It's
 still 256MB (virtual) memeory in total.

You have 128Mb of good healthy RAM and 128MB of safety-net and trash dump.
 In other words, even WITH swap you dont solve the problem, you only postpone 
 any
 problems until later, AND you lose some flash/sd storage room for swap 
 permanently.

You have 128Mb of memory to wait for the problem to solve itself or 
better to do something about it.
 It only makes sense to allocate swap if you know beforehand you'll need more 
 than the
 available RAM.
Wrong.
No one plans to use swap. Using swap is bad. (for most cases)
But not having swap is far worse.

And come on. Software is not perfect. Sometimes we have to live with a 
dreamteam like (old) firefox and x11. I had times when they had both 
hundreds of megs virtual mem. But everything was fine because it all was 
just harmlessly been swaped away. I restarted them every weekend to not 
let it become worse.
Not ideal, but should the system rather be unusable in this condition?

-- 
Drucken Sie diese Mail bitte nur auf Recyclingpapier aus.
Please print this mail only on recycled paper.

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: ASU - out of memory?

[Steven Kurylo]

 And come on. Software is not perfect. Sometimes we have to live with a
 dreamteam like (old) firefox and x11. I had times when they had both
 hundreds of megs virtual mem. But everything was fine because it all was
 just harmlessly been swaped away. I restarted them every weekend to not
 let it become worse.
 Not ideal, but should the system rather be unusable in this condition?

You're assuming the system will be usable when an application
misbehaves and 50mb gets swapped out.  On a desktop, sure your points
are valid.

I'm not sure this is true on Freerunner.  None of the embedded systems
I've used have had swap.  What happens when you haven't received a
call for several hours and the application you're using forces it to
swap out?  Can you still answer a call in time?

I'd rather see a smart oom killer which will only kill non-essential
applications.  Adding 128mb of swap just pushes the problem back and
slows down the entire phone.

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: ASU - out of memory?

[Sander van Grieken]

On Thursday 21 August 2008 19:33:24 Steven Kurylo wrote:
  And come on. Software is not perfect. Sometimes we have to live with a
  dreamteam like (old) firefox and x11. I had times when they had both
  hundreds of megs virtual mem. But everything was fine because it all was
  just harmlessly been swaped away. I restarted them every weekend to not
  let it become worse.
  Not ideal, but should the system rather be unusable in this condition?

 You're assuming the system will be usable when an application
 misbehaves and 50mb gets swapped out.  On a desktop, sure your points
 are valid.

 I'm not sure this is true on Freerunner.  None of the embedded systems
 I've used have had swap.  What happens when you haven't received a
 call for several hours and the application you're using forces it to
 swap out?  Can you still answer a call in time?

Exactly.

 I'd rather see a smart oom killer which will only kill non-essential
 applications.  Adding 128mb of swap just pushes the problem back and
 slows down the entire phone.

For a phone, the algorithm could be as simple as killing the process that has 
allocated the most memory. The essential system services and the basic UI 
applications usually have a small footprint, and the biggest consumer of 
memory is most likely a leaky UI app that's not part of the main system 
anyway.

For a production server with large databases this doesn't work of course, but 
there you're already in big trouble if you have to fallback on the 
oom-killer.

Sander


___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: ASU - out of memory?

[Clinton Ebadi]

Carsten Haitzler (The Rasterman) [EMAIL PROTECTED] writes:

 really - we need a userspace oom that is Smarter (it knows what a system
 daemon is and what a user application is and what is a necessary user desktop
 process), so it will always kill apps not the phone daemon or the window
 manager or the launcher etc. it will stick to finding visible apps that have
 gone rogue and nuke them. the userspace oom can run as root - mlock() its
 memoryspace in and give itself a realtime priority with setsched(). every
 second or so - check the memory state of the system, if it begins to get low -
 issue a warning signal to processes (could use dbus or something more basic
 like system signals - use SIGURG or something pretty much unused these days).
 any well behaved app when it gets such a signal should:

 * free any memory it doesnt REALLY need (any caches or any data loaded that it
 can get back from a file again), so save out stuff and free.
 * if it does garbage collection - collect now. asap!
 * if it makes sense - exit the process altogether (if it is really not a
 useful process and it can be started later when things get better)
[...]
 yes.. this requires work on the part of app authors in userspace. but it's the
 right thing to do (imho).

You've just rediscovered one of the few good design decisions of the
l4hurd project. See the bits on memory allocation in [0]. 'Tis a shame
that Hurd has pretty much failed (l4hurd and ngHurd got closer and
closer to fixing the terrible flaws of Mach Hurd ... but, alas, l4
stagnated and Coyotos has yet to appear dooming us to another 30 years
of UNIX it seems).

[0] 
http://www.walfield.org/papers/20070104-walfield-access-decomposition-policy-refinement.pdf

-- 
Corinne: rub a dub dub nekked in the tub

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: ASU - out of memory?

[Esben Stien]

Carsten Haitzler (The Rasterman) [EMAIL PROTECTED] writes:

 and luckily those smart fellas in kernel developer land.. made
 kernel overcommit.. a tunable parameter! and... cunningly.. on the
 FR (and as wel on my desktop) it's turned off! :) so... a moot point
 really. :)

I was more thinking of applications filling up RAM and then things
will have to be paged out if you don't want things to crash/get
killed. Something is definitely wrong if you're actually using the
swap area on this device, but things like this are observable with
vmstat. Unfortunately, you can't plug in more memory when you see that
your need exceeds the capacity of the device;).

A 128Mio space of memory is not that huge, especially if you want to
use it for something cool and you will at some points pass this mark
and things will die; given the current algorithm it might not be
something you want to get killed.

-- 
Esben Stien is [EMAIL PROTECTED] s  a 
 http://www. s tn m
  irc://irc.  b  -  i  .   e/%23contact
   sip:b0ef@   e e 
   jid:b0ef@n n

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: ASU - out of memory?

[Tilman Baumann]

Am 21.08.2008 um 19:33 schrieb Steven Kurylo:

 And come on. Software is not perfect. Sometimes we have to live  
 with a
 dreamteam like (old) firefox and x11. I had times when they had both
 hundreds of megs virtual mem. But everything was fine because it  
 all was
 just harmlessly been swaped away. I restarted them every weekend to  
 not
 let it become worse.
 Not ideal, but should the system rather be unusable in this  
 condition?

 You're assuming the system will be usable when an application
 misbehaves and 50mb gets swapped out.  On a desktop, sure your points
 are valid.

Remember, linux swaps unused mem first.


 I'm not sure this is true on Freerunner.  None of the embedded systems
 I've used have had swap.
Because they where really embedded.
Openmoko is more or less a mobile desktop.
 What happens when you haven't received a
 call for several hours and the application you're using forces it to
 swap out?  Can you still answer a call in time?

 I'd rather see a smart oom killer which will only kill non-essential
 applications.  Adding 128mb of swap just pushes the problem back and
 slows down the entire phone.

Slowing down is clearly better than instant crashing.
How ever any early warning system for low memory looks, it can not  
guarantee it will be able to fix thing before mem runs out.
This is also true with swap, but much much less probable.

Can you answer al call? I don't know - probably.
Besides, i can live with bad service while my phone is recovering from  
a nearby crash.
And just imagine. Your phone is in a fragile low memory state, and a  
call comes in and the phone app needs to be launched.
Everything great? Sure not.  No matter how.

Though i don't think it will suffer too much from swapping. If i need  
to, i will test this too.

And you are talking like i like to use the swap all the time. No!
I just want it to be there before it is to late in case memory gets low.

Regards
  Tilman

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: ASU - out of memory?

[Steven Kurylo]

 I'm not sure this is true on Freerunner.  None of the embedded systems
 I've used have had swap.
 Because they where really embedded.
 Openmoko is more or less a mobile desktop.

Its embedded because of the limited resources available to it.

 Slowing down is clearly better than instant crashing.
 How ever any early warning system for low memory looks, it can not
 guarantee it will be able to fix thing before mem runs out.
 This is also true with swap, but much much less probable.

 Can you answer al call? I don't know - probably.
 Besides, i can live with bad service while my phone is recovering from
 a nearby crash.
 And just imagine. Your phone is in a fragile low memory state, and a
 call comes in and the phone app needs to be launched.
 Everything great? Sure not.  No matter how.

The dialer app would be exempt from the killer.

There are two issues here.  First, you want a smart oom killer
regardless if you have swap or not.  Core services would never been
killed.

Secondly, is it better to have very slow swap to prolong oom killer
being launched?  You could do extensive testing to change my mind, but
I think swap is going to provide a horrible user experience.  Its
going to take forever to swap 50mb off the SD card.  I'd much rather
always have top speed and less applications running.

But really we can't even be sure there will be an SD card, so that
makes most of this moot.  Now a patch could be made to scan for swap
space on start up, so people like you who want it can insert an SD
card with swap defined.

As for an new oom killer, I think it should expose an file in sysfs.
We can then watch that file, if its 0 everything is ok.  If its 4,
we're running low on memory, less than 10%?,  and our daemon
monitoring this could send the user a popup telling them to close
something.  When there is less than 1%, the file changes to 9.  The
userspace daemon then kills a non critical service.  Something along
those lines, we need a kernel guy to let us know whats possible.

Or perhaps the sysfs file should be 0 for normal or 1 for start
killing.  Then leave it up to userspace daemon to figure out if we're
at send a pop up to the user stage.

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: ASU - out of memory?

[The Rasterman]

On Thu, 21 Aug 2008 14:29:45 -0400 Clinton Ebadi [EMAIL PROTECTED]
babbled:

 You've just rediscovered one of the few good design decisions of the
 l4hurd project. See the bits on memory allocation in [0]. 'Tis a shame
 that Hurd has pretty much failed (l4hurd and ngHurd got closer and
 closer to fixing the terrible flaws of Mach Hurd ... but, alas, l4
 stagnated and Coyotos has yet to appear dooming us to another 30 years
 of UNIX it seems).
 
 [0]
 http://www.walfield.org/papers/20070104-walfield-access-decomposition-policy-refinement.pdf

you know what's sad? WinCE even has low on memory messages - standard system
messages like everything else that applications are meant to honor and try and
clean up stuff when they get it. it of course is voluntary - and it's a measure
to avoid HAVING to kill off apps... but even WinCE has it... and we have no
such ability right now - no standard... so we are stuck with uglier hacks as
the only solution. :(

-- 
Carsten Haitzler (The Rasterman) [EMAIL PROTECTED]

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: ASU - out of memory?

[The Rasterman]

On Fri, 22 Aug 2008 00:07:12 +0200 Esben Stien [EMAIL PROTECTED] babbled:

 Carsten Haitzler (The Rasterman) [EMAIL PROTECTED] writes:
 
  and luckily those smart fellas in kernel developer land.. made
  kernel overcommit.. a tunable parameter! and... cunningly.. on the
  FR (and as wel on my desktop) it's turned off! :) so... a moot point
  really. :)
 
 I was more thinking of applications filling up RAM and then things
 will have to be paged out if you don't want things to crash/get
 killed. Something is definitely wrong if you're actually using the
 swap area on this device, but things like this are observable with
 vmstat. Unfortunately, you can't plug in more memory when you see that
 your need exceeds the capacity of the device;).
 
 A 128Mio space of memory is not that huge, especially if you want to
 use it for something cool and you will at some points pass this mark
 and things will die; given the current algorithm it might not be
 something you want to get killed.

128m is massive. it's unimaginably huge. if you need more you need to look hard
at what it is you are doing, and how. seriously. for the kind of processing
power, storage ability and display ability of this device... - and no. not
saying 640k is enough for everyone. i am saying that 128m is a lot. people
who don't think so are incredibly spoilt. possibly are bad programmers who need
to learn more about memory management or should have their programming licenses
revoked! :) (ok just kidding there). but seriously - it's a lot. this is why
instead of running off to swap for help - which just overflows your ram problems
onto disk (and sooner or later swap fills - then what? you haven't fixed the
original problem - just delayed it).

-- 
Carsten Haitzler (The Rasterman) [EMAIL PROTECTED]

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: ASU - out of memory?

[The Rasterman]

On Thu, 21 Aug 2008 18:40:52 +0200 Tilman Baumann [EMAIL PROTECTED] babbled:

 Carsten Haitzler (The Rasterman) wrote:
  On Thu, 21 Aug 2008 16:44:36 +0200 Tilman Baumann [EMAIL PROTECTED]
  babbled:
  
  Carsten Haitzler (The Rasterman) wrote:
  On Thu, 21 Aug 2008 17:50:26 +0200 Esben Stien [EMAIL PROTECTED]
  babbled:
 
  Tilman Baumann [EMAIL PROTECTED] writes:
 
  all the linux memory overcommit behaviour more or less depends on
  the fact that it can allways save it's ass by using swap. (Instead
  of helplessley crashing)
  Yes, or killing the application. Not having swap is nonsense;). If you
  are using swap something is wrong, right, but then you fix it. I find
  it strange that the debian install didn't make a little swap
  partition.
  and luckily those smart fellas in kernel developer land.. made kernel
  overcommit.. a tunable parameter! and... cunningly.. on the FR (and as wel
  on my desktop) it's turned off! :) so... a moot point really. :)
  pardon? Honestly? This is absurd!
  Why? I don't get it.
 
  I mean, how did you get the impression that overcomitting is a bad thing?
  
  when it makes:
  
  
  myptr = malloc(somesize);
  if (myptr == NULL) return -1; // return and unwind
  // continue as normal
  
  
  useless so i mayaswell do:
  
  myptr = malloc(somesize);
  if (myptr == NULL) abort();
  
  
  ala glib. all that error checking code programs have becomes stupid. i
  should just assume all mallocs succeed!. as they likely will and part way
  through accessing the memory malloc actually returned - i may segv because
  i dont have enough ram. that's just bad! what's the point of error codes
  and returns then? whats the point of even trying to handle an error? :( may
  as well throw in the towel ala glib.
  
 
 Not being able to malloc memory and not having any physical memory left 
 are just two sepereate things. At least on modern linux systems.

no they are not. i write code.

myptr = malloc(mysize);

it returns failuer (NULL) then i need to deal with it.
it returns a pointer - it succeeded. i asked for memory - it gave it to me. the
problem is with overcommit success returns are lies. they MAY have the memory -
they may not. part way through using the pages it returned and SAID i could
have, it can just segv - as it is overcommitted and out of pages. this means
that suddenly return values can't be trusted for memory allocations anymore.
any attempts to handle NULL returns may as well not exist as success cases can
be undetectable failures. it's just a stupid policy. sure - its there to work
around stupid userspace code that goes off allocing massive blobs of ram that
it then never goes and uses, but the kernel should go punish all apps for this
- those apps being stupid should be punished and have their code fixed.

 Memory overcommit saves (physical) memory. And not just a bit.
 And with some swap safety net it is reasonably save to do so.
 
 And how should it be better if _any_ app gets knocked out by running in 
 the memory limit? Usually it can't help the situation. Especially if 
 some other app that is eating all memory.
 The effect is more or less the same. Some random poor app has to be 
 killed. (Since suicide is often the only way to react to malloc fails)
 
 Turning overcommit off is in my eyes only a poor 'bury one's head in the 
 sand' solution which effectively does improve nothing.

i disagree. overcommit is a bury head in sand solution. it means you just go
and avoid the original problem of allocing much more memory than you really
need.

 Overcommit and swap is just the winner. Everyone does it so. And in my 
 eyes rightly so. It is fast and efficient.
 And even without swap, i would not want to turn overcommit off.
 
 Swap and overcommit is just the dream team. Fast and efficient memory. 
 And if something goes wrong, you have plenty of time (and mem) for 
 solving the problem.
 Contrary to without, because how could you fix any low memory condition 
 with not allocating any more memory? Driving something into swap to be 
 able to do something about it is just right.
 
 And just if you thought so. Overcommit is not just lazy don't care for 
 errors behaviour but a really smart optimisation.

wrong. it means i can no longer trust a successful malloc() calloc() realloc()
or even alloca() return. ever. the return of a valid pointer which according to
the manuals provided ever since for these calls, could be an error state too.
and thew only way to handle it is have your own sigsegv/bus handlers and on a
segv which accesses a known allocated chunk of memory, realise that overcommit
just screwed you and try and save yourself (and frankly by this stage - you
may as well just segv, exit or try re-exec yourself freshly).

is is not elegant. it is not good. it is a cute hack that makes badly-behaved
apps not impact things as much. it just sticks the head in the sand and doesn't
go and fix the apps. it means the kernel tries to pick up the badly bloated

Re: ASU - out of memory?

[The Rasterman]

On Thu, 21 Aug 2008 19:54:26 +0200 Sander van Grieken [EMAIL PROTECTED] 
babbled:

 On Thursday 21 August 2008 19:33:24 Steven Kurylo wrote:
   And come on. Software is not perfect. Sometimes we have to live with a
   dreamteam like (old) firefox and x11. I had times when they had both
   hundreds of megs virtual mem. But everything was fine because it all was
   just harmlessly been swaped away. I restarted them every weekend to not
   let it become worse.
   Not ideal, but should the system rather be unusable in this condition?
 
  You're assuming the system will be usable when an application
  misbehaves and 50mb gets swapped out.  On a desktop, sure your points
  are valid.
 
  I'm not sure this is true on Freerunner.  None of the embedded systems
  I've used have had swap.  What happens when you haven't received a
  call for several hours and the application you're using forces it to
  swap out?  Can you still answer a call in time?
 
 Exactly.
 
  I'd rather see a smart oom killer which will only kill non-essential
  applications.  Adding 128mb of swap just pushes the problem back and
  slows down the entire phone.
 
 For a phone, the algorithm could be as simple as killing the process that has 
 allocated the most memory. The essential system services and the basic UI 
 applications usually have a small footprint, and the biggest consumer of 
 memory is most likely a leaky UI app that's not part of the main system 
 anyway.
 
 For a production server with large databases this doesn't work of course, but 
 there you're already in big trouble if you have to fallback on the 
 oom-killer.

true - and the kernel oom killer should mostly handle this, BUT it is possible
to do better. a userspace oom killer can trawl all .desktop files and thus know
if the app is run by a user (base on command), or if its a system app that can
be run, or if its installed later etc. etc.

such a userspace oom isn't hard to do. it's pretty simple.


-- 
Carsten Haitzler (The Rasterman) [EMAIL PROTECTED]

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community