Bug#496771: Deb AMD64 eats huge amounts of memory (and babies?) because of badly built libs
On Wed, Aug 27, 2008 at 12:50:45PM +0200, Gustaf R??ntil?? wrote: > Package: general > Version: AMD64 > > This is basically a debian AMD64-version of the bug report for ubuntu > AMD64 bug 24691 [1]. > > The problem is (seems to be) that a lot of libraries are built with > alignment above 2**3. Most of these cases are actually 2**20 in ubuntu > AMD64 and 2**21 in debian AMD64. In other words, 1 and 2 MB > correspondingly! > > I often see loose and vague arguments such as "if 99MB of that is > shared, the calculator is really only 'using' 1MB of ram -- and that's > fine" [2]. > It's not fine. And it's certainly incorrect. Just because a library is > shared doesn't mean it's fine that it consumes megabytes(!) of memory > in vain. Especially libraries that are "shared" "between" 1 process. > Now, I can't figure out why such huge amounts of memory is hogged on > my computer. But I need to restart X about once a week. If I don't, my > 4 GB of RAM is quickly filled and my 4 GB swap starts to work (hurray, > 15 second delay when changing virtual desktop). It's been like this > since I bought this machine (soon 2 years ago), and I frequently > update my X driver (-radeon, -radeonhd, fglrx, etc), so I doubt they > are to blame for stealing my memory, even though it could've been a > good guess. > > memstat reports lots and lots of libraries which consumes slightly > more than 2 mb each: > $ memstat | grep '\.so' | grep -v PID > gives me 551 lines, and by just grasping the result, easily 90% of > them are slightly more than 2 mb (2**21 + small stuff). > > If these libraries could be built with 2**3 (8 byte) alignment, > instead of 2**21 (2 megabytes), I assume, just like the discussion in > [1], that at least some memory wouldn't be wasted in vain. > So how many libraries (on my system) are built with 2**21 alignment? > > /lib: $ for file in `\ls *.so.*` ; do if objdump -x $file | grep -q -e > '2\*\*21' ; then echo $file ; fi ; done | wc > 99 > /usr/lib: $ for file in `\ls *.so.*` ; do if objdump -x $file | grep > -q -e '2\*\*21' ; then echo $file ; fi ; done | wc > 2777 > /usr/lib/*: $ for file in `\ls */*.so.*` ; do if objdump -x $file | > grep -q -e '2\*\*21' ; then echo $file ; fi ; done | wc > 396 > > 99 + 2777 + 396 = 3272. > Quite a lot of libraries. Loading them all would require roughly 7 GB. > Remember: loading. Not using. And these are just on my system, it's > not even close to "all" libraries in debian. > > Just looking at how much memory pidgin-specific plugins consumes is > frightening: > memstat | grep -E '(purple|pidgin)+.*\.so.*' > Returns 88 libraries, ALL consuming slightly more than 2 mb. How many > of them are "shared" with any other program than pidgin? I'd say none. > But that's just a guess. > So this means at least 200 mb memory usage for pidgin alone? Could > this really be the case?! > $ ps aux | grep pidgin > gustaf 30432 0.2 1.7 631196 68532 ?SAug26 2:36 pidgin > Oh yeah, 631192 kB virtual and 68532 kB resident. > > I say this again, some people argues; "but most of that 616 MB is > shared so it doesn't matter". It matters, because Linux prefers to > swap it, to give place for IO buffers, and when things are being > swapped, holy moses, Ctrl+Alt+Backspace is thy saviour. > Pidgin consuming 616 MB virtual memory is just.. Well.. Messed up. To > put it lightly. > > It could be reasonable to see the rest of the memory hoggers on my > system, to make it clear that this really is a big problem. > > Please note that pidgin is only on 8th place! This is 'top' sorted by > 'M' (memory usage): > > 31095 gustaf20 0 1314m 700m 35m R 30 17.8 252:06.36 firefox-bin > 32197 gustaf20 0 1006m 439m 39m S 12 11.2 167:03.50 epiphany-browse > 30134 root 20 0 625m 256m 14m S8 6.5 99:41.51 Xorg > 2680 gustaf20 0 479m 103m 25m S0 2.6 4:07.31 banshee-1 > 30305 gustaf20 0 438m 82m 17m S0 2.1 0:31.87 /usr/lib/ontv/o > 1621 gustaf20 0 304m 76m 14m S0 1.9 0:42.91 gnome-terminal > 6070 clamav20 0 92524 76m 380 S0 1.9 0:00.00 clamd > 30432 gustaf20 0 616m 66m 28m S1 1.7 2:37.08 pidgin Firefox and epiphany are complete pigs when it comes to memory use. On amd64 machines not running those things look absolutely fine. Firefox causes plenty of swapping on i386 as well. > $ free -m > total used free sharedbuffers cached > Mem: 3934 3891 43 0331 1186 > -/+ buffers/cache: 2373 1561 > Swap: 3859 5 3853 > > Real memory usage of my very recently booted machine (I haven't > started even a small subset of the apps I usually run): 2373 MB! Firefox probably consumes over 50% of the ram of all the applications you are likely to run. Memory is allocated in 4KB pages (since that is what the hardware supports unless you think 2MB pages are a good idea, or 1GB pages). What
Bug#496771: Deb AMD64 eats huge amounts of memory (and babies?) because of badly built libs
Gustaf Räntilä, le Wed 27 Aug 2008 12:50:45 +0200, a écrit : > I haven't seen many discussions about this in debian, and just 1 about ubuntu > [1]. It could be that I have gotten things completely the wrong way (then > please tell me so), but it could also be that this has been overseen for too > long, and now needs serious attention. Do not mistake virtual memory for resident memory. Aligning stuff in the address space is not a problem, it does _not_ consume any memory (even less if the library is not even used). > 31095 gustaf20 0 1314m 700m 35m R 30 17.8 252:06.36 firefox-bin > 32197 gustaf20 0 1006m 439m 39m S 12 11.2 167:03.50 epiphany-browse Here are the suckers, not pidgin. Do not have a look at virtual numbers, as the name says these are virtual. Have a look at the %MEM or RES columns instead. Samuel -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#496771: Deb AMD64 eats huge amounts of memory (and babies?) because of badly built libs
Package: general Version: AMD64 This is basically a debian AMD64-version of the bug report for ubuntu AMD64 bug 24691 [1]. The problem is (seems to be) that a lot of libraries are built with alignment above 2**3. Most of these cases are actually 2**20 in ubuntu AMD64 and 2**21 in debian AMD64. In other words, 1 and 2 MB correspondingly! I often see loose and vague arguments such as "if 99MB of that is shared, the calculator is really only 'using' 1MB of ram -- and that's fine" [2]. It's not fine. And it's certainly incorrect. Just because a library is shared doesn't mean it's fine that it consumes megabytes(!) of memory in vain. Especially libraries that are "shared" "between" 1 process. Now, I can't figure out why such huge amounts of memory is hogged on my computer. But I need to restart X about once a week. If I don't, my 4 GB of RAM is quickly filled and my 4 GB swap starts to work (hurray, 15 second delay when changing virtual desktop). It's been like this since I bought this machine (soon 2 years ago), and I frequently update my X driver (-radeon, -radeonhd, fglrx, etc), so I doubt they are to blame for stealing my memory, even though it could've been a good guess. memstat reports lots and lots of libraries which consumes slightly more than 2 mb each: $ memstat | grep '\.so' | grep -v PID gives me 551 lines, and by just grasping the result, easily 90% of them are slightly more than 2 mb (2**21 + small stuff). If these libraries could be built with 2**3 (8 byte) alignment, instead of 2**21 (2 megabytes), I assume, just like the discussion in [1], that at least some memory wouldn't be wasted in vain. So how many libraries (on my system) are built with 2**21 alignment? /lib: $ for file in `\ls *.so.*` ; do if objdump -x $file | grep -q -e '2\*\*21' ; then echo $file ; fi ; done | wc 99 /usr/lib: $ for file in `\ls *.so.*` ; do if objdump -x $file | grep -q -e '2\*\*21' ; then echo $file ; fi ; done | wc 2777 /usr/lib/*: $ for file in `\ls */*.so.*` ; do if objdump -x $file | grep -q -e '2\*\*21' ; then echo $file ; fi ; done | wc 396 99 + 2777 + 396 = 3272. Quite a lot of libraries. Loading them all would require roughly 7 GB. Remember: loading. Not using. And these are just on my system, it's not even close to "all" libraries in debian. Just looking at how much memory pidgin-specific plugins consumes is frightening: memstat | grep -E '(purple|pidgin)+.*\.so.*' Returns 88 libraries, ALL consuming slightly more than 2 mb. How many of them are "shared" with any other program than pidgin? I'd say none. But that's just a guess. So this means at least 200 mb memory usage for pidgin alone? Could this really be the case?! $ ps aux | grep pidgin gustaf 30432 0.2 1.7 631196 68532 ?SAug26 2:36 pidgin Oh yeah, 631192 kB virtual and 68532 kB resident. I say this again, some people argues; "but most of that 616 MB is shared so it doesn't matter". It matters, because Linux prefers to swap it, to give place for IO buffers, and when things are being swapped, holy moses, Ctrl+Alt+Backspace is thy saviour. Pidgin consuming 616 MB virtual memory is just.. Well.. Messed up. To put it lightly. It could be reasonable to see the rest of the memory hoggers on my system, to make it clear that this really is a big problem. Please note that pidgin is only on 8th place! This is 'top' sorted by 'M' (memory usage): 31095 gustaf20 0 1314m 700m 35m R 30 17.8 252:06.36 firefox-bin 32197 gustaf20 0 1006m 439m 39m S 12 11.2 167:03.50 epiphany-browse 30134 root 20 0 625m 256m 14m S8 6.5 99:41.51 Xorg 2680 gustaf20 0 479m 103m 25m S0 2.6 4:07.31 banshee-1 30305 gustaf20 0 438m 82m 17m S0 2.1 0:31.87 /usr/lib/ontv/o 1621 gustaf20 0 304m 76m 14m S0 1.9 0:42.91 gnome-terminal 6070 clamav20 0 92524 76m 380 S0 1.9 0:00.00 clamd 30432 gustaf20 0 616m 66m 28m S1 1.7 2:37.08 pidgin $ free -m total used free sharedbuffers cached Mem: 3934 3891 43 0331 1186 -/+ buffers/cache: 2373 1561 Swap: 3859 5 3853 Real memory usage of my very recently booted machine (I haven't started even a small subset of the apps I usually run): 2373 MB! $ uptime 12:27:24 up 21:44, 13 users, load average: 1.09, 1.05, 0.99 I'm not sure my mobo supports much more than what I have, but I'm seriously thinking of spending (too much) money on 8 or 16 GB RAM, just to have a system that is usable. Oh yeah, I'm using 'sid', maybe that's important to know, by I certainly doubt it. I haven't seen many discussions about this in debian, and just 1 about ubuntu [1]. It could be that I have gotten things completely the wrong way (then please tell me so), but it could also be that this has been overseen for too long, and now needs serious attention. Memory is cheap nowadays, but it isn't free. And if we're expected to buy do
