Re: Query about failure of Debian 6 64 bit to swap properly
On 8/30/2012 12:28 AM, Bret Busby wrote: [snip] Last night, it was taking up to 20 minutes for the system to respond to a mouse click, and, typing in the text in composing an email message ... and, about 40-60% of the characters that are typed in, simply disappear, requiring composing an ... So, this 64 bit Debian 6 appears to be of the nature, in its status,similar to the nature of the version named experimental. ... You've got some problems alright, a myriad of symptoms you've described. But these symptoms have nothing to do with the way the kernel is swapping to disk, unless you've improperly changed the values of parameters in /proc/sys/vm. You mentioned changing one of them to 70. If you've changed others, change them all back to default values and see if that helps. Three other possibilities come to mind. One, you have a hardware problem with the disk controller or the drive. Check dmesg for errors. Two, a worm has infiltrated your system, and is wreaking having. And/or 3, you're running some applications/processes that execute at boot, causing all of these problems. Are you running any distributed.net programs that make heavy use of CPU/memory/disk? If their process priority was changed (reniced) that could explain a lot of these symptoms. Though I doubt this is the case. Any way you slice it, you've provided no log snippets throughout this thread, and with this kind of system behavior, you'd surely see something in dmesg or syslog pointing to the problem. Do you know how to access your log files? Have you ever used a bash prompt or is your Linux experience limited to the GUI? If the latter, I can see why you're having such problems. If that's the case you are currently incapable of helping us help you. -- Stan -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/50449e39.50...@hardwarefreak.com
Re: Query about failure of Debian 6 64 bit to swap properly
Am Mittwoch, 29. August 2012 schrieb Bret Busby: I do hope that Debian 7 implements memory paging, or swapping. I'm not completely sure what you mean by this :-? It seems to have stopped working properly, in about Debian 5, and I hope that Debian 7 gets it working again. In Debian 5, I could sometimes kickstart memory swapping, by running something like the GIMP, and opening images, then closing the application, at which stage, memory swapping would sometimes start (on a different computer - Debian 5 would not run on this computer), but I have not yet managed to get memory swapping working properly in the 64 bit Debian 6. I do not remember whether the memory swapping works on the 32 bit installation of Debian 6, on my NX5000 laptop. Bret, whatever your issues are, I am very sure that it isn´t that Debian is anable to swap or page memory. If you have 8 GiB of RAM and it swaps another 8 GiB it will be slow. And thats just physics. Harddisks are slow. Much memory paged to to harddisk based swap is slow. Its as easy as that. So the main question is: What on earth uses up 16 GiB or memory on your machine? Solve this and you are likely done with the issue. I have 8 GiB of RAM and the machine rarely swaps when I have two KDE sessions open with a ton of applications and then I have never seen more than 2-3 GiB of swap space used. Granted thats with Debian Sid. Also a ThinkPad T42 with 2 GiB doesn´t swap very much, neither a T23. On the T23 since Sarge in each and every Debian release and lots of interim package versions since then. -- Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201209022123.11104.mar...@lichtvoll.de
Re: Query about failure of Debian 6 64 bit to swap properly
Am Mittwoch, 29. August 2012 schrieb Bob Proulx: Then press F6 to change the sort function. Use the up and down cursor keys to select VIRT for sorting by size of virtual memory usage. What programs are the top virtual memory consumers on your system? (On mine it is usually firefox.) Based upon what those memory hogs are on your system we can advise what action might be taken. No! Virtual memory size does not say a single thing about real physical memory usage. An application can happily allocate 1 GiB or more of virtual memory without every having the Linux kernel allocating physical memory at all except for the management of the memory allocation at all. Virtual memory is just address space. Unless the application writes to it, the kernel does nothing, repeat, nothing in physical memory. So its resident set size. And even that is not accurate, cause it usually collects libary memory usage as well. So when you have 100 processes using libc6 the library is still in memory just once. So an approach is account to the process the resident set size of the library divided by the amount of processes that use it. In newer KDE versions the process monitor has a detailed memory statistics page that does just that. I never have seen that in a shell command up to know. -- Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201209022129.07797.mar...@lichtvoll.de
Re: Query about failure of Debian 6 64 bit to swap properly
Am Mittwoch, 29. August 2012 schrieb Bret Busby: Then press F6 to change the sort function. Use the up and down cursor keys to select VIRT for sorting by size of virtual memory usage. What programs are the top virtual memory consumers on your system? (On mine it is usually firefox.) Based upon what those memory hogs are on your system we can advise what action might be taken. opera web browser. Each window of it shows as using 14GB of virtual memory. Again: That does say nothing except for that Opera tends to be quite greedy regarding allocating virtual address space. But then on a 64 Bit system there is plenty of it. On this ThinkPad T520 with Sandybridge i5 Dualcore: martin@merkaba:~ grep VmallocTotal /proc/meminfo VmallocTotal: 34359738367 kB If I am not completely mistaken thats 32768 GiB. Look for RSS or resident set size as virtual address space is just the kernels way to play poker with applications. Its the physically used memory that counts. -- Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201209022135.34544.mar...@lichtvoll.de
Re: Query about failure of Debian 6 64 bit to swap properly
Am Mittwoch, 29. August 2012 schrieb Bret Busby: Hello. Hello Bret, In the ongoing saga of the inability of the 64 bit version of Debian 6 to swap properly, so that an i3 CPU with 8GB of RAM and a 40GB swap partition, runs about as fast as an 8086 trying to run MS Windows 3, a possible cause has occurred to me. […] So, my query is this; is the inability of 64 bit Debian 6, to swap properly, instead using increasing amounts of RAM until it runs out of RAM, then crashing, while having 40GB of unused swap partition allocated and swappiness set to 70, due to the inability of the file manager to cope with filesize greater than 1GB? 1) Debian is not unable to swap. Period. Neither Lenny, nor Squeeze, nor Wheezy, nor Sid. 2) So your issue must be something else. As already pointed out later in thread it seems to be insane amount of memory usage. 3) 40 GB of swap with 8 GB of RAM is an quite insane setup. A harddisk takes in about 40-100 MiB per second on sequential access. Swap accesses can be quite random. So its at least 10 seconds per GiB of swap or 80 seconds per 8 GiB of swap. Usually *lots* more. And writing can be even slower. The machine is likely to start to crawl to a halt by anything near to 16 GiB of swap usage, likely even before depending on storage speed. Thats physics. 4) Any filemanager I have yet seen in Debian is able to copy with files bigger than 1 GiB. FAT32 cannot take files larger than 4 GiB with default cluster sizes. Some applications may still have problems with really huge files. (See Large File Support in linux kernel and large / huge file support in Ext4.) 5) You say you are inable to handle files bigger than 1 GiB. What *exactly* happens? What are the error message. Please stay just by the facts. Preliminary conclusions can be completely of track. In a situation of starting memory pressure I ask you to: - free -m - or better cat /proc/meminfo - vmstat 1 and at least 30-40 lines of it Additionally please install and then do: - Start atop - Press m - Look which processes have the highest RGROW and SWAPSZ values It seems to my atop sorts by resident set usage which seems saner to me than sorting by virtual memory usage. Also look for anything that atop marks by turquoise or especial red color. In in doubt try to crap some text snapshots of it. Should be quite easy since by default it has a 10 second delay between updates. Ciao, -- Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201209022152.42384.mar...@lichtvoll.de
Re: Query about failure of Debian 6 64 bit to swap properly
Darac Marjal wrote: However, as you've noted, once you run out of RAM, the kernel should start moving the less-frequently used pages into Swap. In theory, the OOM-killer should only come into play when both are full. Agreed. However, I can see a couple of situations where that may not happen: Note that the Bret said he was running on a 64-bit machine and that he had configured 40G of swap space. Therefore he won't be experiencing 32-bit memory limitations and he won't be experiencing out of virtual memory space. Instead a large process will simply grow larger. If the process walks through its own memory with any regularity then it will all be somewhat active which will cause it all to page in and out of swap space. That will definitely cause the machine to behave very slowly while it is actively swapping memory pages. Bret Busby wrote: It used to work, much better, with Debian 3 and 3.1; I can't remember much about Debian 4, then, as previously mentioned, I had the problem and the solution as such, with Debian 5, and, now, with Debian 6, memory management appears to simply not work, making Debian 6, at least in the 64 bit version, of the nature of the attributes used to describe the experimental version of Debian. But remember that amd64 is a recent addition to Debian. Older Debian 3 and 3.1 did not have an amd64 64-bit version. (There were 64-bit alpha versions and so on.) Therefore it is likely that you were running a 32-bit version of the system back at that time. That is a critical difference. It is critical because then you would have been capped out at 2G per process (or 3G if they linked it -N non-shared to get that extra gig) and by that limitation. If you were running a 32-bit version of Opera it would have run out of address space for any more ram and could not possibly have grown to be a 14G process image as you found it to be recently. It is only possible now because you have a 64-bit system now. As far as the swap space goes... I imagine that you probably increased it due to the virtual memory pressure from Opera's large process size. Since it has been growing to that large size it is using up virtual memory. I assume that is why you increased to that large size of swap. Which is fine. Because otherwise being out of virtual memory the linux kernel would have activated the out-of-memory killer and it would have killed processes on your machine until it could pay its memory overcommit debt. (Depending upon the setting of vm.overcommit_memory.) I don't know why Opera is so large on your system. I think addressing it would be best place to improve your situation. Bob signature.asc Description: Digital signature
Re: Query about failure of Debian 6 64 bit to swap properly
Martin Steigerwald wrote: schrieb Bob Proulx: Then press F6 to change the sort function. Use the up and down cursor keys to select VIRT for sorting by size of virtual memory usage. What programs are the top virtual memory consumers on your system? (On mine it is usually firefox.) Based upon what those memory hogs are on your system we can advise what action might be taken. No! Virtual memory size does not say a single thing about real physical memory usage. :-) Note that I was crafting my answer so as to be most useful to the original poster trying to understand the problem. It may not have been the right answer for 100% of all situations. But I think in this situation it was still very useful. And it did appear to have been a successful way to diagnose the 14G problem. :-) An application can happily allocate 1 GiB or more of virtual memory without every having the Linux kernel allocating physical memory at all except for the management of the memory allocation at all. Yes. Depending upon the setting of vm.overcommit_memory. But with the default value of memory overcommit being enabled that is true. Of course the default value isn't enterprise quality reliable and so I always disable it. But I know that unless you have hit this problem before and have researched it and made the decision to change it that the default value of overcommit is what is in effect. Virtual memory is just address space. Unless the application writes to it, the kernel does nothing, repeat, nothing in physical memory. Yes. I completely agree with what you are saying. I understand it very well. :-) So its resident set size. And even that is not accurate, cause it usually collects libary memory usage as well. Right. Neither virtual memory size nor resident set size is the 100% correct way to tell what is happening. But I didn't think it was efficient to give a long lecture on memory hierarchy at that point in time in the email message. Instead I simply tried to give a way to diagnose the problem in a way that was useful to the original poster. This is just a general problem of phone/email support. If I am trapped on a research station in the antarctic and am required to perform an emergency appendectomy while in communication with a real surgeon then I am really hoping that the doctor who is guiding me through it will take into consideration the situation and will give me practical instructions for that moment and won't require teaching me a multiple year medical residency. The appendectomy patient may not survive that long otherwise. :-) So I understand that you are appalled by how inaccurate my advice was from a pedantically correct point of view. (How could I have possibly said such a thing!) Believe me that I was fully aware of it. Just the same I think it was the most efficient way to find the problem. Sometimes life isn't always about being 100% correct. Sometimes one must be practical about things. So when you have 100 processes using libc6 the library is still in memory just once. So an approach is account to the process the resident set size of the library divided by the amount of processes that use it. In newer KDE versions the process monitor has a detailed memory statistics page that does just that. I never have seen that in a shell command up to know. It would be awesome if there were a script or other tool that could help automate this debugging for the user. Perhaps you might write one? Because I am pretty sure that if you were to try to describe the steps needed to 9 out of 10 typical random users they wouldn't be motivated to actually do it. Instead they would rather reboot. The KDE process monitor sounds interesting. But I dread the idea of installing all of the KDE environment to play with it. Perhaps if I remember the next time I already have a need for KDE otherwise I will give it a try. Bob signature.asc Description: Digital signature
Re: Query about failure of Debian 6 64 bit to swap properly
have a similar system to yours and just downloaded the 64 bit Opera on Squeeze. It installed quickly and with no errors. I have now ran it for several hours pushing it harder than I normally push my Iceweasel 16 and have experienced none of the issues you describe. Its lightning fast, stable, and uses very little CPU. Here is the Opera line from TOP: PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 20 0 601m 278m 33m S 0 3.5 0:55.05 opera As you can see, it is using 0% CPU and 3.5% mem, which is significant, but here is the Iceweasel line: 20 0 919m 266m 40m S 0 3.3 4:59.08 firefox-bin So a very negligible difference. In conclusion, I think this testing indicates that you are either running a corrupted install of Opera or you are running a poorly written/broken extension. On Wed, 29 Aug 2012 04:23:47 -0400, Bret Busby b...@busby.net wrote: Hello. In the ongoing saga of the inability of the 64 bit version of Debian 6 to swap properly, so that an i3 CPU with 8GB of RAM and a 40GB swap partition, runs about as fast as an 8086 trying to run MS Windows 3, a possible cause has occurred to me. I have found that the file manager, despite the operating system being 64 bit, cannot cope with a filesize of about 1GB. I can download and delete files with a filesize of up to about 1.2GB, but, if I try to move or copy them, the system crashes, and requires the power button to be held down, to strangle the system, to shut the system down. Sometimes, when this happens, I have to reboot a few times to get the system going again. I am thus wondering whether, with the swap partition beng 40GB, and thus, bigger than the about 1GB maximum filesize that the file manager can cope with, the inability to use the swap partition, is due to the failure of the file manager to cope with files over about 1GB. I can download, copy, and move, files with file size of about 600-700GB, such as CD ISO files, but not much bigger. So, my query is this; is the inability of 64 bit Debian 6, to swap properly, instead using increasing amounts of RAM until it runs out of RAM, then crashing, while having 40GB of unused swap partition allocated and swappiness set to 70, due to the inability of the file manager to cope with filesize greater than 1GB? I do hope that Debian 7 implements memory paging, or swapping. Thank you in anticipation. -- Bret Busby Armadale West Australia .. So once you do know what the question actually is, you'll know what the answer means. - Deep Thought, Chapter 28 of Book 1 of The Hitchhiker's Guide to the Galaxy: A Trilogy In Four Parts, written by Douglas Adams, published by Pan Books, 1992 -- Using Opera's revolutionary email client: http://www.opera.com/mail/ -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/op.wj046pbzer0...@debian-dave.home
Re: Query about failure of Debian 6 64 bit to swap properly
On Wed, Aug 29, 2012 at 10:07:12PM +0100, Lisi wrote: On Wednesday 29 August 2012 21:15:01 Claudius Hubig wrote: A problem that I (appear to) have found, is that the malware named javascript appears to cause havoc in continually increasing usage of RAM. Javascript is a programming language, not a malware. He knows that. He is expressing his opinion of Javascript. Or he might be confusing java with javascript (LOL, while ducking) -- If you're not careful, the newspapers will have you hating the people who are being oppressed, and loving the people who are doing the oppressing. --- Malcolm X -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120831222049.GG12794@tal
Re: Query about failure of Debian 6 64 bit to swap properly
On Thu, Aug 30, 2012 at 01:28:05PM +0800, Bret Busby wrote: On Wed, 29 Aug 2012, Bob Proulx wrote: [cut] I enable javascript in Opera, as I use it for most online financial transactions, including online banking, and, of the (more?) major web browsers, as far as I am aware, opera is the only one that has not yet been breached as far as security is concerned. I have seen multiple CERT advisories for the Mozilla and Microsoft web browsers. Be aware that fewer advisories may or may not be a good thing here. Opera is proprietary (closed source) so the only way to test it for security problems is to attack the compiled browser. Admittedly, this is the same with Internet Explorer, though but that has a higher market share. [cut[ I had understood that the operating system (in the case or Linux, the kernel?) controls memory management, so that, depending on the settings, once a threshold, for example, 50% of RAM, is used, the operating system would start paging memory, using the allocated swap space, to provide system stability until both swap space and RAM are totally used, then crash, rather than just using up all of the RAM and mostly ignoring the swap space and crashing the system, without significantly using the swap space. No. The Linux philosophy is that you paid good money to have all that nice, fast RAM in your system so why not make use of it. There will usually be a certain amount of RAM free, but the kernel prefers to keep things like buffers, disk caches etc filling your RAM rather than needlessly discarding them (If you discard cached data and need it again, you have to take the hit of reading from disk. But if you need the RAM for something else such as an application, then THAT's the time to discard some cached data). However, as you've noted, once you run out of RAM, the kernel should start moving the less-frequently used pages into Swap. In theory, the OOM-killer should only come into play when both are full. However, I can see a couple of situations where that may not happen: * A single application (Opera, in this case) is requesting more memory than you have RAM. In this case, the rest of the system is swapped out (hence the hideous responsiveness) but not using all of your swap, and eventually the kernel says I'm sorry Dave * You are on a 32-bit system and an application has requested more memory than the kernel can address (3-4GB depending on kernel options). I am apparently wrong. It used to work, much better, with Debian 3 and 3.1; I can't remember much about Debian 4, then, as previously mentioned, I had the problem and the solution as such, with Debian 5, and, now, with Debian 6, memory management appears to simply not work, making Debian 6, at least in the 64 bit version, of the nature of the attributes used to describe the experimental version of Debian. As this appears to be a problem with Opera, have you considered raising the issue with Opera's Support (http://www.opera.com/support/)? signature.asc Description: Digital signature
Re: Query about failure of Debian 6 64 bit to swap properly
On Thu, 30 Aug 2012 02:16:23 +0800, Bret Busby wrote: On Wed, 29 Aug 2012, Camaleón wrote: (...) Some hints: - With 8 GiB of RAM you can (almost) safely turn off your swap at all, it shouldn't be used. You can indeed run this test (→ turn off swap) to see how your system behaves. - The kernel will use all of the system resources which are available and that includes /swap. The problem is that the computer runs out of RAM. Then you have to find the culprit. Did you looked at the logs? The RAM usage increases, until it runs out of RAM, then, as at present, the system becomes morbidly slow, and takes a few seconds to respond to key presses or mouse moves, then, after a while, it just crashes. That's the normal behaviour when you run/hit OOM. After about 95% of the RAM is used, so that the computer becomes frustratingly slow, it starts to use the swap space, up to about the same amount as the RAM, which is about 1/6 of the swap space. Example: at present, the SystemMNonitor shows Memory usage - 7.6GB (98.4%) of 7.7GB Swap usage - 7.9GB (19.3%) of 40.9GB and my XT with 640KB RAM and a 10MB HDD, used to run faster than this is running. Fine, but what's the process that is taking all of your RAM resources? Second, you say you can't delete big files (1 GiB of size) because your system becomes unmanageable and runs out of memory. This is of course not normal (even a system with as little as 256 MiB of RAM shouldn't experience this problem at all). No. I said that I can save and delete files up to about 1.2GB. I can not save files larger than about 1.2GB, to the system. That's what I said :-? The file manager crashes, and, crashes the system, when the saved file size gets to 1.2GB, if it gets that big. I have had some attempted file saves crash at 12MB, crashing the system. When the file manager crashes, do you have anything (message, warning...) printed in the screen? The file manager does not work well. To debug a problem with your file manager run the test I told you (use the command line to create/delete big files). If command line commands are working well, try with different file manager (you can be hitting some kind of specific bug for that specific application). I do hope that Debian 7 implements memory paging, or swapping. I'm not completely sure what you mean by this :-? It seems to have stopped working properly, in about Debian 5, and I hope that Debian 7 gets it working again. What exactly do you think it has been stopped from working proplery? In Debian 5, I could sometimes kickstart memory swapping, by running something like the GIMP, and opening images, then closing the application, at which stage, memory swapping would sometimes start (on a different computer - Debian 5 would not run on this computer), but I have not yet managed to get memory swapping working properly in the 64 bit Debian 6. I do not remember whether the memory swapping works on the 32 bit installation of Debian 6, on my NX5000 laptop. I don't think you have a problem with swapping but experiencing some kind of OOM problem which leaves your system exhaust. Remember that your swap will be used if you have configured the system (kernel) for doing so. Anyway, unless you really need it for something specific (and if so, I would like to know what), 40 GiB of /swap is a bit insane and a waste of space :-) Greetings, -- Camaleón -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/k1npkr$pns$3...@ger.gmane.org
Query about failure of Debian 6 64 bit to swap properly
Hello. In the ongoing saga of the inability of the 64 bit version of Debian 6 to swap properly, so that an i3 CPU with 8GB of RAM and a 40GB swap partition, runs about as fast as an 8086 trying to run MS Windows 3, a possible cause has occurred to me. I have found that the file manager, despite the operating system being 64 bit, cannot cope with a filesize of about 1GB. I can download and delete files with a filesize of up to about 1.2GB, but, if I try to move or copy them, the system crashes, and requires the power button to be held down, to strangle the system, to shut the system down. Sometimes, when this happens, I have to reboot a few times to get the system going again. I am thus wondering whether, with the swap partition beng 40GB, and thus, bigger than the about 1GB maximum filesize that the file manager can cope with, the inability to use the swap partition, is due to the failure of the file manager to cope with files over about 1GB. I can download, copy, and move, files with file size of about 600-700GB, such as CD ISO files, but not much bigger. So, my query is this; is the inability of 64 bit Debian 6, to swap properly, instead using increasing amounts of RAM until it runs out of RAM, then crashing, while having 40GB of unused swap partition allocated and swappiness set to 70, due to the inability of the file manager to cope with filesize greater than 1GB? I do hope that Debian 7 implements memory paging, or swapping. Thank you in anticipation. -- Bret Busby Armadale West Australia .. So once you do know what the question actually is, you'll know what the answer means. - Deep Thought, Chapter 28 of Book 1 of The Hitchhiker's Guide to the Galaxy: A Trilogy In Four Parts, written by Douglas Adams, published by Pan Books, 1992 -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/alpine.deb.2.00.1208291605240.11...@bret-dd-workstation.busby.net
Re: Query about failure of Debian 6 64 bit to swap properly
On Wed, 29 Aug 2012 16:23:47 +0800, Bret Busby wrote: (...) So, my query is this; is the inability of 64 bit Debian 6, to swap properly, instead using increasing amounts of RAM until it runs out of RAM, then crashing, while having 40GB of unused swap partition allocated and swappiness set to 70, due to the inability of the file manager to cope with filesize greater than 1GB? I think you are talking about two problems here. Let's see... First, it seems that you have some sort of problems with your swap but, what are those problems exactly? Some hints: - With 8 GiB of RAM you can (almost) safely turn off your swap at all, it shouldn't be used. You can indeed run this test (→ turn off swap) to see how your system behaves. - The kernel will use all of the system resources which are available and that includes /swap. Second, you say you can't delete big files (1 GiB of size) because your system becomes unmanageable and runs out of memory. This is of course not normal (even a system with as little as 256 MiB of RAM shouldn't experience this problem at all). Some hints: - How are you deleting the files? You can try with a GUI based application (e.g., nautillus) and then compare the results using the command line (rm my_big_file). - Are you getting any entry for the OOM error at the logs (/var/log/ syslog)? I do hope that Debian 7 implements memory paging, or swapping. I'm not completely sure what you mean by this :-? Greetings, -- Camaleón -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/k1ldkl$koe$9...@ger.gmane.org
Re: Query about failure of Debian 6 64 bit to swap properly
On Wed, 29 Aug 2012, Camaleón wrote: On Wed, 29 Aug 2012 16:23:47 +0800, Bret Busby wrote: (...) So, my query is this; is the inability of 64 bit Debian 6, to swap properly, instead using increasing amounts of RAM until it runs out of RAM, then crashing, while having 40GB of unused swap partition allocated and swappiness set to 70, due to the inability of the file manager to cope with filesize greater than 1GB? I think you are talking about two problems here. Let's see... First, it seems that you have some sort of problems with your swap but, what are those problems exactly? Some hints: - With 8 GiB of RAM you can (almost) safely turn off your swap at all, it shouldn't be used. You can indeed run this test (→ turn off swap) to see how your system behaves. - The kernel will use all of the system resources which are available and that includes /swap. The problem is that the computer runs out of RAM. The RAM usage increases, until it runs out of RAM, then, as at present, the system becomes morbidly slow, and takes a few seconds to respond to key presses or mouse moves, then, after a while, it just crashes. After about 95% of the RAM is used, so that the computer becomes frustratingly slow, it starts to use the swap space, up to about the same amount as the RAM, which is about 1/6 of the swap space. Example: at present, the SystemMNonitor shows Memory usage - 7.6GB (98.4%) of 7.7GB Swap usage - 7.9GB (19.3%) of 40.9GB and my XT with 640KB RAM and a 10MB HDD, used to run faster than this is running. Second, you say you can't delete big files (1 GiB of size) because your system becomes unmanageable and runs out of memory. This is of course not normal (even a system with as little as 256 MiB of RAM shouldn't experience this problem at all). No. I said that I can save and delete files up to about 1.2GB. I can not save files larger than about 1.2GB, to the system. The file manager crashes, and, crashes the system, when the saved file size gets to 1.2GB, if it gets that big. I have had some attempted file saves crash at 12MB, crashing the system. The file manager does not work well. I do hope that Debian 7 implements memory paging, or swapping. I'm not completely sure what you mean by this :-? It seems to have stopped working properly, in about Debian 5, and I hope that Debian 7 gets it working again. In Debian 5, I could sometimes kickstart memory swapping, by running something like the GIMP, and opening images, then closing the application, at which stage, memory swapping would sometimes start (on a different computer - Debian 5 would not run on this computer), but I have not yet managed to get memory swapping working properly in the 64 bit Debian 6. I do not remember whether the memory swapping works on the 32 bit installation of Debian 6, on my NX5000 laptop. -- Bret Busby Armadale West Australia .. So once you do know what the question actually is, you'll know what the answer means. - Deep Thought, Chapter 28 of Book 1 of The Hitchhiker's Guide to the Galaxy: A Trilogy In Four Parts, written by Douglas Adams, published by Pan Books, 1992
Re: Query about failure of Debian 6 64 bit to swap properly
On Thu, Aug 30, 2012 at 02:16:23AM +0800, Bret Busby wrote: On Wed, 29 Aug 2012, Camaleón wrote: On Wed, 29 Aug 2012 16:23:47 +0800, Bret Busby wrote: (...) So, my query is this; is the inability of 64 bit Debian 6, to swap properly, instead using increasing amounts of RAM until it runs out of RAM, then crashing, while having 40GB of unused swap partition allocated and swappiness set to 70, due to the inability of the file manager to cope with filesize greater than 1GB? I think you are talking about two problems here. Let's see... I agree with Camaleó. You have at least two problems. The problem is that the computer runs out of RAM. OK, why? Run top and tap 'm'. You will see the processes in your system ordered according to memory usage. What are the top offenders, and how much are they using? I can not save files larger than about 1.2GB, to the system. The file manager crashes, and, crashes the system, when the saved file size gets to 1.2GB, if it gets that big. I have had some attempted file saves crash at 12MB, crashing the system. Which file manager are you using? There are roughly 300 of them available in Debian. Does the same problem occur when you rm a file from the command line? In Debian 5, I could sometimes kickstart memory swapping, by running something like the GIMP, and opening images, then closing the application, at which stage, memory swapping would sometimes start (on a different computer - Debian 5 would not run on this computer), but I have not yet managed to get memory swapping working properly in the 64 bit Debian 6. I do not remember whether the memory swapping works on the 32 bit installation of Debian 6, on my NX5000 laptop. This is bizarre. All you need to do to start swap availability is to have a swap partition or swap file created and identified in /etc/fstab. /sbin/swapon -s will show you what partitions or files you are using, and how big they are and how much is used. Your goal is to generally not be using swap at all. You can turn swapping on and off with /sbin/swapon -a /sbin/swapoff -a -a is for all. -dsr- -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120829183605.gf4...@randomstring.org
Re: Query about failure of Debian 6 64 bit to swap properly
Bret Busby wrote: The problem is that the computer runs out of RAM. The RAM usage increases, until it runs out of RAM, then, as at present, the system becomes morbidly slow, and takes a few seconds to respond to key presses or mouse moves, then, after a while, it just crashes. What you are describing here is not a failure of Debian to swap properly (really the Linux kernel which is a component). But instead you are describing a process or set of processes that have a memory leak and that are consuming ram without bounds. That is NOT NORMAL. Find those processes and take corrective action. After about 95% of the RAM is used, so that the computer becomes frustratingly slow, it starts to use the swap space, up to about the same amount as the RAM, which is about 1/6 of the swap space. Yes. That is the way that Unix-like systems work and have for the last forty years. And why we always try to avoid thrashing swap space. Example: at present, the SystemMNonitor shows Memory usage - 7.6GB (98.4%) of 7.7GB Swap usage - 7.9GB (19.3%) of 40.9GB What is consuming 7.9G of swap? That is very large and very unusual. Find that and fix it. Do nothing else until you understand where the memory is going. and my XT with 640KB RAM and a 10MB HDD, used to run faster than this is running. Of course. Any system that is thrashing will be much slower than it should be running. You know the old joke about, doctor, it hurts when I do this, doctor says, don't do that? Same thing here. Don't do that. Second, you say you can't delete big files (1 GiB of size) because your system becomes unmanageable and runs out of memory. This is of course not normal (even a system with as little as 256 MiB of RAM shouldn't experience this problem at all). No. I said that I can save and delete files up to about 1.2GB. I can not save files larger than about 1.2GB, to the system. The file manager crashes, and, crashes the system, when the saved file size gets to 1.2GB, if it gets that big. I have had some attempted file saves crash at 12MB, crashing the system. The file manager does not work well. I agree that this sounds like a separate problem. But I find it strange that both problems exist together on a system. So they are probably related somehow. But concentrate on one first and the solution to it may also solve the other. I do hope that Debian 7 implements memory paging, or swapping. Debian practically means Linux. Linux *does* implement memory paging and swapping. I am sorry if your particular system is broken in some way. But I assure you that it is something that you have done to your system and that behavior is not normal. No one other than yourself is seeing the problem you are seeing. Therefore no one else can debug it for you. I'm not completely sure what you mean by this :-? It seems to have stopped working properly, in about Debian 5, and I hope that Debian 7 gets it working again. I assure you that it is working on Debian 5, 6, and 7. In Debian 5, I could sometimes kickstart memory swapping, by running something like the GIMP, and opening images, then closing the application, at which stage, memory swapping would sometimes start (on a different computer - Debian 5 would not run on this computer), but I have not yet managed to get memory swapping working properly in the 64 bit Debian 6. I do not remember whether the memory swapping works on the 32 bit installation of Debian 6, on my NX5000 laptop. You are creating a superstition instead of working it through. That won't help. Instead find out what is using all of your virtual memory. The tool I like the best is 'htop'. Install it. It is nice and I think you will like it. # apt-get install htop Then run it: $ htop Then press F6 to change the sort function. Use the up and down cursor keys to select VIRT for sorting by size of virtual memory usage. What programs are the top virtual memory consumers on your system? (On mine it is usually firefox.) Based upon what those memory hogs are on your system we can advise what action might be taken. To the list... Does anyone have any nice 'ps' recipies for doing the same thing? Bob signature.asc Description: Digital signature
Re: Query about failure of Debian 6 64 bit to swap properly
On Wed, 29 Aug 2012, Dan Ritter wrote: On Thu, Aug 30, 2012 at 02:16:23AM +0800, Bret Busby wrote: On Wed, 29 Aug 2012, Camaleón wrote: On Wed, 29 Aug 2012 16:23:47 +0800, Bret Busby wrote: (...) snip /sbin/swapon -s will show you what partitions or files you are using, and how big they are and how much is used. Your goal is to generally not be using swap at all. /sbin/swapon -s FilenameType Size Used Priority /dev/sda7 partition 42860340 8379428 -1 -- Bret Busby Armadale West Australia .. So once you do know what the question actually is, you'll know what the answer means. - Deep Thought, Chapter 28 of Book 1 of The Hitchhiker's Guide to the Galaxy: A Trilogy In Four Parts, written by Douglas Adams, published by Pan Books, 1992
Re: Query about failure of Debian 6 64 bit to swap properly
On Wed, Aug 29, 2012 at 01:01:28PM -0600, Bob Proulx wrote: To the list... Does anyone have any nice 'ps' recipies for doing the same thing? ps STUFF --sort vsize -dsr- -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120829193828.gg4...@randomstring.org
Re: Query about failure of Debian 6 64 bit to swap properly
On Wed, 29 Aug 2012, Bob Proulx wrote: Bret Busby wrote: The problem is that the computer runs out of RAM. snip The tool I like the best is 'htop'. Install it. It is nice and I think you will like it. # apt-get install htop Then run it: $ htop Then press F6 to change the sort function. Use the up and down cursor keys to select VIRT for sorting by size of virtual memory usage. What programs are the top virtual memory consumers on your system? (On mine it is usually firefox.) Based upon what those memory hogs are on your system we can advise what action might be taken. opera web browser. Each window of it shows as using 14GB of virtual memory. A problem that I (appear to) have found, is that the malware named javascript appears to cause havoc in continually increasing usage of RAM. Some web sites use client-side processing, via javascript, and I regard it as malicious, and I believe that a well written web site should not use client-side processing, but should instead use server-side processing. In some web browsers that I use, I have javascript disabled, but I left it enabled in opera. -- Bret Busby Armadale West Australia .. So once you do know what the question actually is, you'll know what the answer means. - Deep Thought, Chapter 28 of Book 1 of The Hitchhiker's Guide to the Galaxy: A Trilogy In Four Parts, written by Douglas Adams, published by Pan Books, 1992 -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/alpine.deb.2.00.1208300352220.18...@bret-dd-workstation.busby.net
Re: Query about failure of Debian 6 64 bit to swap properly
Hello Bret, Bret Busby b...@busby.net wrote: opera web browser. Each window of it shows as using 14GB of virtual memory. Nice. Opera usually uses as much memory as it sees fit, but you can set the memory cache manually (Preferences → Advanced → History). A very wild guess would be that due to your extremely large swap space (which is rarely used by anything), Opera thinks it might use much more memory than normally. On my system (8 GB RAM + 8 GB swap), Opera uses something between 1 and 3 GB (residual/virtual), depending on how long it runs. A problem that I (appear to) have found, is that the malware named javascript appears to cause havoc in continually increasing usage of RAM. Javascript is a programming language, not a malware. Some web sites use client-side processing, via javascript, and I regard it as malicious, and I believe that a well written web site should not use client-side processing, but should instead use server-side processing. This is simply wrong, since many things are much faster with additional client-side processing, not to mention the fact that one may specifically want to do some client-side processing instead of trusting the server with everything (and needing a TCP round-trip for each request…). In some web browsers that I use, I have javascript disabled, but I left it enabled in opera. I suggest you take a close look at the pages you usually visit. Best regards, Claudius -- A board is the planck unit of boredom. http://chubig.net telnet nightfall.org 4242 signature.asc Description: PGP signature
Re: Query about failure of Debian 6 64 bit to swap properly
On Wednesday 29 August 2012 21:15:01 Claudius Hubig wrote: A problem that I (appear to) have found, is that the malware named javascript appears to cause havoc in continually increasing usage of RAM. Javascript is a programming language, not a malware. He knows that. He is expressing his opinion of Javascript. Lisi -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201208292207.12303.lisi.re...@gmail.com
Re: Query about failure of Debian 6 64 bit to swap properly
Bret Busby wrote: opera web browser. Each window of it shows as using 14GB of virtual memory. Yowsa! So if you exit Opera as a test then suddenly a lot of memory is freed up and the system is suddenly back to its normal speedy state? Any other processes hiding behind it that are the second tier of memory hog? A problem that I (appear to) have found, is that the malware named javascript appears to cause havoc in continually increasing usage of RAM. Yes. The curse of the modern world. I normally run Firefox with the noscript extension. Then for the web sites that require Javascript I use either Chromium or Midori with everything enabled, access the site, then exit the browser afterward to free up the memory resources. Some web sites use client-side processing, via javascript, and I regard it as malicious, and I believe that a well written web site should not use client-side processing, but should instead use server-side processing. The term you are looking for is Progressive Enhancement. http://en.wikipedia.org/wiki/Progressive_Enhancement As opposed to Graceful Degradation. Which is a terrible problem. But one which more and more people are creating every day. In some web browsers that I use, I have javascript disabled, but I left it enabled in opera. The outside world does what the outside world does. If you can't find a way to limit its memory use then you might just not be able to run Opera continuously for long periods of time without restarting it. Bob -- They called me mad, and I called them mad, and damn them, they outvoted me. --Nathaniel Lee signature.asc Description: Digital signature
Re: Query about failure of Debian 6 64 bit to swap properly
On Wed, 29 Aug 2012, Bob Proulx wrote: Bret Busby wrote: opera web browser. Each window of it shows as using 14GB of virtual memory. Yowsa! So if you exit Opera as a test then suddenly a lot of memory is freed up and the system is suddenly back to its normal speedy state? Any other processes hiding behind it that are the second tier of memory hog? A problem that I (appear to) have found, is that the malware named javascript appears to cause havoc in continually increasing usage of RAM. Yes. The curse of the modern world. I normally run Firefox with the noscript extension. Then for the web sites that require Javascript I use either Chromium or Midori with everything enabled, access the site, then exit the browser afterward to free up the memory resources. I enable javascript in Opera, as I use it for most online financial transactions, including online banking, and, of the (more?) major web browsers, as far as I am aware, opera is the only one that has not yet been breached as far as security is concerned. I have seen multiple CERT advisories for the Mozilla and Microsoft web browsers. Other browsers that I use, include iceweasel and iceape and konqueror, and I have used galeon and one that I think is named Epithany, and found opera to be the most stable of them. The problem with the progressive consumption of RAM, happens with any web browser that I run, that has javascript enabled. I have also found that none of the web browsers implement the stop all unwanted pop-ups, when that switch is set. Unwanted pop-ups still occur. I have found australian government web sites that use javascript, including the ABC (Australian Broadcasting Corporation) news web site, and online television guides that use javascript, to be bad for what they do with the javascript. I had understood that the operating system (in the case or Linux, the kernel?) controls memory management, so that, depending on the settings, once a threshold, for example, 50% of RAM, is used, the operating system would start paging memory, using the allocated swap space, to provide system stability until both swap space and RAM are totally used, then crash, rather than just using up all of the RAM and mostly ignoring the swap space and crashing the system, without significantly using the swap space. I am apparently wrong. It used to work, much better, with Debian 3 and 3.1; I can't remember much about Debian 4, then, as previously mentioned, I had the problem and the solution as such, with Debian 5, and, now, with Debian 6, memory management appears to simply not work, making Debian 6, at least in the 64 bit version, of the nature of the attributes used to describe the experimental version of Debian. It has just taken me about 35 minutes to be able to log in to the system, and, logging in is blind; the screen is black, and, after typing in the password, blind, it takes up to about 35 minutes for the system to respond. Last night, it was taking up to 20 minutes for the system to respond to a mouse click, and, typing in the text in composing an email message (I am using alpine, the replacement for pine), most of the typing is blind, as characters take a while to be displayed, and, about 40-60% of the characters that are typed in, simply disappear, requiring composing an email message to take about three times as much times as it should, due to the required patching up of the text that disappears. So, this 64 bit Debian 6 appears to be of the nature, in its status,similar to the nature of the version named experimental. And, someone's response that I seem to be the only person who has these problems with Debian 6, makes me wonder whether it is instead, that I am the only one who has these problems, that has managed, after much time and effort, to be able to contact the outside world and be able to send a help message, indicating what is happening, and thus, that others may have the same problems, but are unable to either log in to their systems, or, to compose and send email messages out. -- Bret Busby Armadale West Australia .. So once you do know what the question actually is, you'll know what the answer means. - Deep Thought, Chapter 28 of Book 1 of The Hitchhiker's Guide to the Galaxy: A Trilogy In Four Parts, written by Douglas Adams, published by Pan Books, 1992 -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/alpine.deb.2.00.1208301259420.24...@bret-dd-workstation.busby.net
