Re: ghc releasing memory during compilation
Bulat Ziganshin wrote: Sunday, March 12, 2006, 8:00:05 PM, you wrote: ghbrho Subject: ghc releasing memory during compilation ghbrho Am I right in thinking that ghc's rts can free memory back to the system ghbrho when the heap pressure reduces (at least if it's doing a compacting GC)? ghbrho In this case if it can do so, it should be quite dramatic. It'd ought to ghbrho be able to go from 400Mb back down to just a few Mb or so. i've suggested the same just 11 months ago. after long discussion Simon Marlow declined my proposal because this will raise gc times in the situations when memory is enough by whole 10%! when i suggested just to check available PHYSICAL memory he answered that he don't know that is physical memory, only virtual memory matters I think that's a mischaracterisation of what I said. Actually I said that I didn't understand your comment about physical vs. virtual memory, and in fact, looking back at the message, I still don't understand it :) http://www.haskell.org//pipermail/glasgow-haskell-users/2005-April/008373.html I think what you're suggesting is that the runtime should detect the amount of physical memory on the system and auto-tune itself to switch to compacting collection when its residency reaches that amount. This is certainly something we could do. Bear in mind that GHC is not necessarily the only process running on the machine, though, and what about running multiple GHCs? Also, you can do this by setting your GHC_RTS environment variable. Suppose you have 1G of physical mem. You want GHC to give up when it reaches 1.5G, and you want to switch to compacting collection when the residency reaches 300M. You could do this: export GHC_RTS='-M1.5G -c20' because 300M is 20% of 1.5G. Perhaps a better interface would be to allow you to specify exactly the residency at which to switch to compaction. Cheers, Simon ___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: ghc releasing memory during compilation
Duncan Coutts wrote: As some are aware some Wash modules cause ghc to take a very large amount of memory. It also generates very large C and assembler files (the .raw_s file for one module is nearly 50Mb). So unsurprisingly it also makes gcc take a very large amount of memory. Unfortunately for people with weaker machines these happen at the same time. That is at the same time that gcc starts taking 300Mb, ghc is still taking up 400Mb. Even on machines with 1Gb of ram this pushes everything else out into swap. Note that unless constrained, ghc will take even more than 400Mb to build wash (I noticed it using over 750Mb). The Gentoo ebuild already limits ghc to 400Mb on 64 bit machines (and 200Mb on 32bit ones). What I was wondering is if ghc could do a major GC and free most of it's memory back to the system just before it calls gcc to compile the .hc code that ghc has generated. That way the memory spike of ghc and gcc would not coincide and our machines would not be brought to a crawl. Am I right in thinking that ghc's rts can free memory back to the system when the heap pressure reduces (at least if it's doing a compacting GC)? No, not at the moment. One thing we planned to do but never got around to is to use madvise() to improve swapping behaviour when memory is tight (see the thread that Bulat refferred to). In this case if it can do so, it should be quite dramatic. It'd ought to be able to go from 400Mb back down to just a few Mb or so. Yes it ought to. A related problem is that the block allocator is really stupid. It makes no attempt to reduce fragmentation, and in fact freeing blocks is O(n) because the freelist is kept sorted in address order. This isn't usually an issue, although it has been reported to be noticeable with very large residencies (500M+). It's on my list to fix at some point. I mention this because freeing all that memory might not be possible if it is highly fragmented. Cheers, Simon ___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: ghc releasing memory during compilation
On Mon, 2006-03-13 at 12:47 +, Simon Marlow wrote: Am I right in thinking that ghc's rts can free memory back to the system when the heap pressure reduces (at least if it's doing a compacting GC)? No, not at the moment. One thing we planned to do but never got around to is to use madvise() to improve swapping behaviour when memory is tight (see the thread that Bulat refferred to). In this case if it can do so, it should be quite dramatic. It'd ought to be able to go from 400Mb back down to just a few Mb or so. Yes it ought to. A related problem is that the block allocator is really stupid. It makes no attempt to reduce fragmentation, and in fact freeing blocks is O(n) because the freelist is kept sorted in address order. This isn't usually an issue, although it has been reported to be noticeable with very large residencies (500M+). It's on my list to fix at some point. I mention this because freeing all that memory might not be possible if it is highly fragmented. Ah, I hoped that if we were using the compacting GC then it might be able to defragment (since it is copying anyway) and thus free large contiguous blocks, eg munmap()ing whole MBlocks. Duncan ___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
ghc releasing memory during compilation
As some are aware some Wash modules cause ghc to take a very large amount of memory. It also generates very large C and assembler files (the .raw_s file for one module is nearly 50Mb). So unsurprisingly it also makes gcc take a very large amount of memory. Unfortunately for people with weaker machines these happen at the same time. That is at the same time that gcc starts taking 300Mb, ghc is still taking up 400Mb. Even on machines with 1Gb of ram this pushes everything else out into swap. Note that unless constrained, ghc will take even more than 400Mb to build wash (I noticed it using over 750Mb). The Gentoo ebuild already limits ghc to 400Mb on 64 bit machines (and 200Mb on 32bit ones). What I was wondering is if ghc could do a major GC and free most of it's memory back to the system just before it calls gcc to compile the .hc code that ghc has generated. That way the memory spike of ghc and gcc would not coincide and our machines would not be brought to a crawl. Am I right in thinking that ghc's rts can free memory back to the system when the heap pressure reduces (at least if it's doing a compacting GC)? In this case if it can do so, it should be quite dramatic. It'd ought to be able to go from 400Mb back down to just a few Mb or so. Duncan ___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
ghc releasing memory during compilation
Sunday, March 12, 2006, 8:00:05 PM, you wrote: ghbrho Subject: ghc releasing memory during compilation ghbrho Am I right in thinking that ghc's rts can free memory back to the system ghbrho when the heap pressure reduces (at least if it's doing a compacting GC)? ghbrho In this case if it can do so, it should be quite dramatic. It'd ought to ghbrho be able to go from 400Mb back down to just a few Mb or so. i've suggested the same just 11 months ago. after long discussion Simon Marlow declined my proposal because this will raise gc times in the situations when memory is enough by whole 10%! when i suggested just to check available PHYSICAL memory he answered that he don't know that is physical memory, only virtual memory matters sorry for my style, but i've proposed rather detailed solution of this problem and i'm discontented what i was not heard at that time :( ... you can find details in thread starting with the following letter: 2) if, for example, program's data before GC is 80 mb and after GC is 60 mb then the program will occupy after GC the whole 140 mb and ALL this space will be marked by OS as used! if there's a memory shortage, old program data even can be swapped to disk despite the fact that we absolutely don't need them! that behaviour significantly enlarge memory needs of GHC-compiled programs if this unused memory will be returned to OS or marked as unneeded after GC then will problem will go on. preferably this must be done during the time of GC, on each page which contents has been already compacted. in this case such program will never use more than 80mb of real memory (+ 1 page + memory for GC) -- Best regards, Bulatmailto:[EMAIL PROTECTED] ___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
