Re: [GHC] #698: GHC's internal memory allocator never releases memory back to the OS
#698: GHC's internal memory allocator never releases memory back to the OS
-+--
Reporter: guest | Owner: igloo
Type: bug | Status: closed
Priority: highest | Milestone: 7.0.1
Component: Runtime System |Version: 6.12.1
Resolution: fixed | Keywords:
Testcase: | Blockedby:
Difficulty: Moderate (less than a day) | Os: Unknown/Multiple
Blocking: | Architecture: Unknown/Multiple
Failure: None/Unknown|
-+--
Changes (by igloo):
* status: new = closed
* resolution: = fixed
Comment:
Fixed by:
{{{
Mon Nov 1 16:18:02 GMT 2010 Ian Lynagh ig...@earth.li
* On Windows, when returning memory to the OS, we try to release it
as well as decommiting it.
}}}
in HEAD and 7.0.
--
Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/698#comment:45
GHC http://www.haskell.org/ghc/
The Glasgow Haskell Compiler
___
Glasgow-haskell-bugs mailing list
Glasgow-haskell-bugs@haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #698: GHC's internal memory allocator never releases memory back to the OS
#698: GHC's internal memory allocator never releases memory back to the OS -+-- Reporter: guest | Owner: igloo Type: bug | Status: new Priority: highest | Milestone: 7.0.1 Component: Runtime System |Version: 6.12.1 Resolution: | Keywords: Testcase: N/A | Blockedby: Difficulty: Moderate (less than a day) | Os: Unknown/Multiple Blocking: | Architecture: Unknown/Multiple Failure: None/Unknown| -+-- Changes (by simonpj): * priority: high = highest Comment: Standard Chartered say it's release critical, and offer to help. -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/698#comment:41 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler ___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #698: GHC's internal memory allocator never releases memory back to the OS
#698: GHC's internal memory allocator never releases memory back to the OS -+-- Reporter: guest | Owner: igloo Type: bug | Status: new Priority: high| Milestone: 7.0.1 Component: Runtime System |Version: 6.12.1 Resolution: | Keywords: Testcase: N/A | Blockedby: Difficulty: Moderate (less than a day) | Os: Unknown/Multiple Blocking: | Architecture: Unknown/Multiple Failure: None/Unknown| -+-- Changes (by simonmar): * owner: = igloo -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/698#comment:40 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler ___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #698: GHC's internal memory allocator never releases memory back to the OS
#698: GHC's internal memory allocator never releases memory back to the OS -+-- Reporter: guest | Owner: Type: bug | Status: new Priority: high| Milestone: 7.0.1 Component: Runtime System |Version: 6.12.1 Resolution: | Keywords: Testcase: N/A | Blockedby: Difficulty: Moderate (less than a day) | Os: Unknown/Multiple Blocking: | Architecture: Unknown/Multiple Failure: None/Unknown| -+-- Changes (by simonmar): * owner: igloo = * status: closed = new * resolution: fixed = Comment: On Win32 we're de-committing memory, but we're not releasing the reserved address space. This is important to some people (Lennart co. at Standard Chartered need this for example) so I'm re-opening the ticket. -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/698#comment:39 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler ___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #698: GHC's internal memory allocator never releases memory back to the OS
#698: GHC's internal memory allocator never releases memory back to the OS
-+--
Reporter: guest | Owner: igloo
Type: bug | Status: closed
Priority: high| Milestone: 6.14.1
Component: Runtime System |Version: 6.12.1
Resolution: fixed | Keywords:
Testcase: N/A | Blockedby:
Difficulty: Moderate (less than a day) | Os: Unknown/Multiple
Blocking: | Architecture: Unknown/Multiple
Failure: None/Unknown|
-+--
Changes (by igloo):
* status: new = closed
* resolution: = fixed
Comment:
Fixed:
{{{
Fri Aug 13 10:04:02 PDT 2010 Ian Lynagh ig...@earth.li
* Return memory to the OS; trac #698
}}}
--
Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/698#comment:38
GHC http://www.haskell.org/ghc/
The Glasgow Haskell Compiler
___
Glasgow-haskell-bugs mailing list
Glasgow-haskell-bugs@haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #698: GHC's internal memory allocator never releases memory back to the OS
#698: GHC's internal memory allocator never releases memory back to the OS -+-- Reporter: guest |Owner: igloo Type: bug | Status: new Priority: high |Milestone: 6.14.1 Component: Runtime System| Version: 6.12.1 Keywords:| Testcase: N/A Blockedby:| Difficulty: Moderate (less than a day) Os: Unknown/Multiple | Blocking: Architecture: Unknown/Multiple | Failure: None/Unknown -+-- Changes (by simonmar): * priority: normal = high -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/698#comment:34 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler ___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #698: GHC's internal memory allocator never releases memory back to the OS
#698: GHC's internal memory allocator never releases memory back to the OS -+-- Reporter: guest |Owner: igloo Type: bug | Status: new Priority: high |Milestone: 6.14.1 Component: Runtime System| Version: 6.12.1 Keywords:| Testcase: N/A Blockedby:| Difficulty: Moderate (less than a day) Os: Unknown/Multiple | Blocking: Architecture: Unknown/Multiple | Failure: None/Unknown -+-- Comment(by zooko): Unfortunately I'm still receiving notifications in my email about this ticket. I've already removed my email address from the Cc: line. If anyone who understands the trac config better could take me off this ticket I would appreciate it. -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/698#comment:35 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler ___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #698: GHC's internal memory allocator never releases memory back to the OS
#698: GHC's internal memory allocator never releases memory back to the OS -+-- Reporter: guest |Owner: igloo Type: bug | Status: new Priority: high |Milestone: 6.14.1 Component: Runtime System| Version: 6.12.1 Keywords:| Testcase: N/A Blockedby:| Difficulty: Moderate (less than a day) Os: Unknown/Multiple | Blocking: Architecture: Unknown/Multiple | Failure: None/Unknown -+-- Changes (by zooko): * cc: zo...@… (added) Comment: Adding myself to Cc: with the intent to then un-add myself again and see if that makes me receive no more mail about this. -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/698#comment:36 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler ___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #698: GHC's internal memory allocator never releases memory back to the OS
#698: GHC's internal memory allocator never releases memory back to the OS -+-- Reporter: guest |Owner: igloo Type: bug | Status: new Priority: high |Milestone: 6.14.1 Component: Runtime System| Version: 6.12.1 Keywords:| Testcase: N/A Blockedby:| Difficulty: Moderate (less than a day) Os: Unknown/Multiple | Blocking: Architecture: Unknown/Multiple | Failure: None/Unknown -+-- Changes (by zooko): * cc: zo...@… (removed) Comment: Removing myself from Cc:. -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/698#comment:37 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler ___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #698: GHC's internal memory allocator never releases memory back to the OS
#698: GHC's internal memory allocator never releases memory back to the OS -+-- Reporter: guest |Owner: igloo Type: bug | Status: new Priority: normal|Milestone: 6.14.1 Component: Runtime System| Version: 6.12.1 Keywords:| Difficulty: Moderate (less than a day) Os: Unknown/Multiple | Testcase: N/A Architecture: Unknown/Multiple | Failure: None/Unknown -+-- Changes (by Remi): * cc: rt...@… (added) -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/698#comment:31 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler ___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #698: GHC's internal memory allocator never releases memory back to the OS
#698: GHC's internal memory allocator never releases memory back to the OS -+-- Reporter: guest |Owner: igloo Type: bug | Status: new Priority: normal|Milestone: 6.14.1 Component: Runtime System| Version: 6.12.1 Keywords:| Difficulty: Moderate (less than a day) Os: Unknown/Multiple | Testcase: N/A Architecture: Unknown/Multiple | Failure: None/Unknown -+-- Changes (by zooko): * cc: zo...@… (added) Comment: I still don't believe that it is important for GHC to release memory back to the operating system during a process, and I'm unsubscribing from this ticket so hopefully I'll hear no more about it. -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/698#comment:32 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler ___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #698: GHC's internal memory allocator never releases memory back to the OS
#698: GHC's internal memory allocator never releases memory back to the OS -+-- Reporter: guest |Owner: igloo Type: bug | Status: new Priority: normal|Milestone: 6.14.1 Component: Runtime System| Version: 6.12.1 Keywords:| Difficulty: Moderate (less than a day) Os: Unknown/Multiple | Testcase: N/A Architecture: Unknown/Multiple | Failure: None/Unknown -+-- Changes (by zooko): * cc: zo...@… (removed) -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/698#comment:33 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler ___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #698: GHC's internal memory allocator never releases memory back to the OS
#698: GHC's internal memory allocator never releases memory back to the OS -+-- Reporter: guest |Owner: igloo Type: bug | Status: new Priority: normal|Milestone: 6.14.1 Component: Runtime System| Version: 6.12.1 Keywords:| Difficulty: Moderate (less than a day) Os: Unknown/Multiple | Testcase: N/A Architecture: Unknown/Multiple | Failure: None/Unknown -+-- Changes (by kfrdbs): * cc: kfr...@… (added) -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/698#comment:30 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler ___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #698: GHC's internal memory allocator never releases memory back to the OS
#698: GHC's internal memory allocator never releases memory back to the OS -+-- Reporter: guest |Owner: igloo Type: bug | Status: new Priority: normal|Milestone: 6.14.1 Component: Runtime System| Version: 6.12.1 Keywords:| Difficulty: Moderate (less than a day) Os: Unknown/Multiple | Testcase: N/A Architecture: Unknown/Multiple | Failure: None/Unknown -+-- Changes (by igloo): * priority: low = normal * milestone: 6.12.3 = 6.14.1 -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/698#comment:29 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler ___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #698: GHC's internal memory allocator never releases memory back to the OS
#698: GHC's internal memory allocator never releases memory back to the OS -+-- Reporter: guest |Owner: igloo Type: bug | Status: new Priority: low |Milestone: 6.12.3 Component: Runtime System| Version: 6.12.1 Keywords:| Difficulty: Moderate (less than a day) Os: Unknown/Multiple | Testcase: N/A Architecture: Unknown/Multiple | Failure: None/Unknown -+-- Changes (by tinlix): * version: 6.4.1 = 6.12.1 Comment: I have the following use case where it is important for the GC to free memory. I have a huge matrix file which needs to be multiplied with a vector. The matrix is 100Gb in size, so it's not possible to use matlab to do it. I used lazy string and thought I could use the readFile in Data.ByteString.Lazy to read in the matrix one row at a time and do the multiplication with the vector. The whole program just does a simple run over the huge file. It turns out the program run out of memory as it goes through the big file. Finally I have to go back to C++/STL which enables me to write a program that takes constant memory and solves the problem fine. IMHO, this bug is serious. What a GC should provide, i.e. constant memory consumption now becomes O(n). In a sense, this makes functions such as readFile useless for serious use. -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/698#comment:27 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler ___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #698: GHC's internal memory allocator never releases memory back to the OS
#698: GHC's internal memory allocator never releases memory back to the OS -+-- Reporter: guest |Owner: igloo Type: bug | Status: new Priority: low |Milestone: 6.12.3 Component: Runtime System| Version: 6.12.1 Keywords:| Difficulty: Moderate (less than a day) Os: Unknown/Multiple | Testcase: N/A Architecture: Unknown/Multiple | Failure: None/Unknown -+-- Comment(by igloo): Replying to [comment:27 tinlix]: I have the following use case where it is important for the GC to free memory. I have a huge matrix file which needs to be multiplied with a vector. The matrix is 100Gb in size, so it's not possible to use matlab to do it. I used lazy string and thought I could use the readFile in Data.ByteString.Lazy to read in the matrix one row at a time and do the multiplication with the vector. The whole program just does a simple run over the huge file. It turns out the program run out of memory as it goes through the big file. Finally I have to go back to C++/STL which enables me to write a program that takes constant memory and solves the problem fine. IMHO, this bug is serious. What a GC should provide, i.e. constant memory consumption now becomes O(n). In a sense, this makes functions such as readFile useless for serious use. From your description, it sounds like your program has a space leak. This ticket is not the problem: it is about the memory usage not dropping after a peak, but you expect the memory usage to be constant, so won't have any peaks. I suggest you explain the problem, including the code, to the [http://www.haskell.org/mailman/listinfo/haskell-cafe haskell-cafe mailing list]. -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/698#comment:28 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler ___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #698: GHC's internal memory allocator never releases memory back to the OS
#698: GHC's internal memory allocator never releases memory back to the OS -+-- Reporter: guest |Owner: igloo Type: bug | Status: new Priority: low |Milestone: 6.12 branch Component: Runtime System| Version: 6.4.1 Keywords:| Difficulty: Moderate (less than a day) Os: Unknown/Multiple | Testcase: N/A Architecture: Unknown/Multiple | Failure: None/Unknown -+-- Changes (by PHO): * cc: p...@… (added) * failure: = None/Unknown -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/698#comment:25 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler ___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #698: GHC's internal memory allocator never releases memory back to the OS
#698: GHC's internal memory allocator never releases memory back to the OS -+-- Reporter: guest |Owner: igloo Type: bug | Status: new Priority: low |Milestone: 6.12 branch Component: Runtime System| Version: 6.4.1 Severity: normal| Resolution: Keywords:| Difficulty: Moderate (1 day) Testcase: N/A | Os: Unknown/Multiple Architecture: Unknown/Multiple | -+-- Changes (by YitzGale): * cc: g...@sefer.org (added) -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/698#comment:23 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #698: GHC's internal memory allocator never releases memory back to the OS
#698: GHC's internal memory allocator never releases memory back to the OS -+-- Reporter: guest |Owner: igloo Type: bug | Status: new Priority: low |Milestone: 6.12 branch Component: Runtime System| Version: 6.4.1 Severity: normal| Resolution: Keywords:| Difficulty: Moderate (1 day) Testcase: N/A | Os: Unknown/Multiple Architecture: Unknown/Multiple | -+-- Comment (by simonmar): Replying to [comment:18 guest]: I originally tried to work around the memory growth for darcs.net by adding the max heap option (the -m option IIRC); turned out that setting an upper bound didn't trigger returns to the OS but segfaults. If you see a segfault, please report it! -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/698#comment:21 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #698: GHC's internal memory allocator never releases memory back to the OS
#698: GHC's internal memory allocator never releases memory back to the OS -+-- Reporter: guest |Owner: igloo Type: bug | Status: new Priority: low |Milestone: 6.12 branch Component: Runtime System| Version: 6.4.1 Severity: normal| Resolution: Keywords:| Difficulty: Moderate (1 day) Testcase: N/A | Os: Unknown/Multiple Architecture: Unknown/Multiple | -+-- Comment (by simonmar): Replying to [comment:17 crutcher]: I want to work on this. It seems that there's little agreement on what the 'right' behavior is, because there are many different execution models for which different behaviors are preferable. It seems we need a means of setting a memory reclamation policy, and plugging in some number of implementations of that policy, with flags to set it. Off the top of my head, I see a few obvious ones: * Never return free memory (the current behavior) * Immediately return free memory (the notional behavior) * Return outstanding free memory on 'flush' events (nice for the dll case?) * Fixed Buffer - return free memory over X, for some buffer size X. * Ratio Buffer - return free memory over R, for some ratio of used memory. And there's this one, which I'd like to be able to play with, but has numerous knobs. * Derivative Ratio Buffer - at time t, estimate the derivative D(t) of memory use, and return free memory over R*D(t+h) for some ration R and time step h. Bear in mind that the actual memory requirements fluctuate over time due to GC activity. When the copying GC is being used, at a major GC we require F*L0+L1 memory, where L0 is the amount of live data at the last GC, L1 is the current live data, and F is the value set by `+RTS -F` (default 2). In practice we'll need a little bit more than this, because we GC the cycle after the limit has been reached. So I suggest that after a major GC we * estimate the amount of memory required at the next GC, assuming live data remains constant, call this M, and add a constant C * release any whole megablocks over this limit Provide a way to set C, and/or define it as a fraction of M. I imagine that C == M would be a reasonable default: keep double the current requirements around just in case. People who want to be frugal with memory could set C == M/3. Programs with wildly varying memory requirements will suffer a performance hit if C is too low. Memory could be released between GCs, but the live data value can only be calculated at a major GC, so it makes most sense to release memory at a major GC. Programs compiled with `-threaded` get an automatic major GC when they're idle (idle time set by `+RTS -I`), programs compiled without `-threaded` will have to call `System.Mem.performGC` to release memory if they intend to go idle. -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/698#comment:22 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #698: GHC's internal memory allocator never releases memory back to the OS
#698: GHC's internal memory allocator never releases memory back to the OS -+-- Reporter: guest |Owner: igloo Type: bug | Status: new Priority: low |Milestone: 6.12 branch Component: Runtime System| Version: 6.4.1 Severity: normal| Resolution: Keywords:| Difficulty: Moderate (1 day) Testcase: N/A | Os: Unknown/Multiple Architecture: Unknown/Multiple | -+-- Comment (by crutcher): I want to work on this. It seems that there's little agreement on what the 'right' behavior is, because there are many different execution models for which different behaviors are preferable. It seems we need a means of setting a memory reclamation policy, and plugging in some number of implementations of that policy, with flags to set it. Off the top of my head, I see a few obvious ones: * Never return free memory (the current behavior) * Immediately return free memory (the notional behavior) * Return outstanding free memory on 'flush' events (nice for the dll case?) * Fixed Buffer - return free memory over X, for some buffer size X. * Ratio Buffer - return free memory over R, for some ratio of used memory. And there's this one, which I'd like to be able to play with, but has numerous knobs. * Derivative Ratio Buffer - at time t, estimate the derivative D(t) of memory use, and return free memory over R*D(t+h) for some ration R and time step h. Thoughts? -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/698#comment:17 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #698: GHC's internal memory allocator never releases memory back to the OS
#698: GHC's internal memory allocator never releases memory back to the OS -+-- Reporter: guest |Owner: igloo Type: bug | Status: new Priority: low |Milestone: 6.12 branch Component: Runtime System| Version: 6.4.1 Severity: normal| Resolution: Keywords:| Difficulty: Moderate (1 day) Testcase: N/A | Os: Unknown/Multiple Architecture: Unknown/Multiple | -+-- Comment (by guest): Fixed buffer would be quite acceptable for the Gitit (and most servers) use case, I think. I originally tried to work around the memory growth for darcs.net by adding the max heap option (the -m option IIRC); turned out that setting an upper bound didn't trigger returns to the OS but segfaults. -- gwern -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/698#comment:18 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #698: GHC's internal memory allocator never releases memory back to the OS
#698: GHC's internal memory allocator never releases memory back to the OS -+-- Reporter: guest |Owner: igloo Type: bug | Status: new Priority: low |Milestone: 6.12 branch Component: Runtime System| Version: 6.4.1 Severity: normal| Resolution: Keywords:| Difficulty: Moderate (1 day) Testcase: N/A | Os: Unknown/Multiple Architecture: Unknown/Multiple | -+-- Comment (by crutcher): I've been talking to a colleague about this, and it seems that we can host essentially all of the other approaches on the 'fixed buffer' approach, by periodically sparking a process to change the buffer size. I need to spend some more time mucking about; I'm using this as a ghc hacking starter project. I'd appreciate any suggestions :) -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/698#comment:19 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #698: GHC's internal memory allocator never releases memory back to the OS
#698: GHC's internal memory allocator never releases memory back to the OS -+-- Reporter: guest |Owner: igloo Type: bug | Status: new Priority: low |Milestone: 6.12 branch Component: Runtime System| Version: 6.4.1 Severity: normal| Resolution: Keywords:| Difficulty: Moderate (1 day) Testcase: N/A | Os: Linux Architecture: Unknown/Multiple | -+-- Changes (by NeilMitchell): * cc: ndmitch...@gmail.com (added) Comment: This issue is much more important now we have dll's. Imagine a program that calls out to a Haskell dll, and then returns some result. If the intermediate computation uses 200Mb of RAM then after returning it's very easy to have 200Mb of heap with virtually nothing in it. The argument that other Haskell bit's will soon reuse that heap is also less valid with dll's, as the main program will not be sharing the same heap. -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/698#comment:15 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #698: GHC's internal memory allocator never releases memory back to the OS
#698: GHC's internal memory allocator never releases memory back to the OS -+-- Reporter: guest |Owner: igloo Type: bug | Status: new Priority: low |Milestone: 6.12 branch Component: Runtime System| Version: 6.4.1 Severity: normal| Resolution: Keywords:| Difficulty: Moderate (1 day) Testcase: N/A | Os: Linux Architecture: Unknown/Multiple | -+-- Comment (by simonmar): Jeremy - I think what you're describing here is not related to freeing of memory in the RTS. The virtual memory used by the trivial program is mostly shared libraries. Try `cat /proc/pid/maps` - on my x86_64/Linux system here, the trivial program without `-threaded` needs 16MB of VM, and with `-threaded` needs 20MB. The shared library mappings account for most of the 16MB, and the extra 4MB with `-threaded` is due to the OS threads which seem to reserve 2MB each. GHC's RTS has only allocated 1MB in each case, and there is no memory to free back to the OS. -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/698#comment:14 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #698: GHC's internal memory allocator never releases memory back to the OS
#698: GHC's internal memory allocator never releases memory back to the OS
-+--
Reporter: guest |Owner: igloo
Type: bug | Status: new
Priority: low |Milestone: 6.12 branch
Component: Runtime System| Version: 6.4.1
Severity: normal| Resolution:
Keywords:| Difficulty: Moderate (1 day)
Testcase: N/A | Os: Linux
Architecture: Unknown/Multiple |
-+--
Comment (by JeremyShaw):
I believe that virtual memory counts against your quota on most VPSes. On
my VPS with a 64MB quota, I have a Haskell-based server which only
requires 7MB RSS (and 5MB of that is SHR), but requires 40MB virtual.
In general, the threaded RTS seems to use a lot of virtual memory. On my
system, the one-liner:
{{{
main = getLine return ()
}}}
compiled with, {{{ghc --make -O2 -threaded -o main}}}
requires less than 1MB of RSS but 20MB of virtual. If (unused) virtual
memory really does count against my quota, then there could be an actual
costs savings involved...
--
Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/698#comment:13
GHC http://www.haskell.org/ghc/
The Glasgow Haskell Compiler___
Glasgow-haskell-bugs mailing list
Glasgow-haskell-bugs@haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #698: GHC's internal memory allocator never releases memory back to the OS
#698: GHC's internal memory allocator never releases memory back to the OS -+-- Reporter: guest |Owner: igloo Type: bug | Status: new Priority: low |Milestone: 6.12 branch Component: Runtime System| Version: 6.4.1 Severity: normal| Resolution: Keywords:| Difficulty: Moderate (1 day) Testcase: N/A | Os: Linux Architecture: Unknown/Multiple | -+-- Comment (by simonmar): Replying to [comment:11 zooko]: If the memory usage continues to grow when a substantially similar task is being performed over and over, then this is a memory leak. That's a separate issue from returning memory to the OS. Exactly. If anyone can demonstrate a leak, please create a separate ticket. Personally I don't think returning memory to the OS is worth the effort. The OS will swap it out anyway when it is unused. (Hint: never look at the virtual memory size on linux. It is a useless and misleading number -- it doesn't correlate with anything that we actually care about.) Which is why it hasn't been addressed so far. But there are people who still care about the absolute virtual size of their processes - perhaps because they want to avoid the swapping, or maybe just because they don't like seeing huge processes in `top`. -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/698#comment:12 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #698: GHC's internal memory allocator never releases memory back to the OS
#698: GHC's internal memory allocator never releases memory back to the OS -+-- Reporter: guest |Owner: igloo Type: bug | Status: new Priority: low |Milestone: 6.12 branch Component: Runtime System| Version: 6.4.1 Severity: normal| Resolution: Keywords:| Difficulty: Moderate (1 day) Testcase: N/A | Os: Linux Architecture: Unknown/Multiple | -+-- Comment (by zooko): each time this happens, the memory usage goes up a bit - even though even last bit of the history will get discarded once the page has been constructed and sent off to the client. (We've checked for memory leaks If the memory usage continues to grow when a substantially similar task is being performed over and over, then this is a memory leak. That's a separate issue from returning memory to the OS. Personally I don't think returning memory to the OS is worth the effort. The OS will swap it out anyway when it is unused. (Hint: never look at the virtual memory size on linux. It is a useless and misleading number -- it doesn't correlate with anything that we actually care about.) -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/698#comment:11 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #698: GHC's internal memory allocator never releases memory back to the OS
#698: GHC's internal memory allocator never releases memory back to the OS -+-- Reporter: guest |Owner: igloo Type: bug | Status: new Priority: low |Milestone: 6.12 branch Component: Runtime System| Version: 6.4.1 Severity: normal| Resolution: Keywords:| Difficulty: Moderate (1 day) Testcase: N/A | Os: Linux Architecture: Unknown/Multiple | -+-- Changes (by asklinge): * cc: asklingenb...@gmail.com (added) -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/698#comment:10 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #698: GHC's internal memory allocator never releases memory back to the OS
#698: GHC's internal memory allocator never releases memory back to the OS -+-- Reporter: guest |Owner: igloo Type: bug | Status: new Priority: low |Milestone: 6.12 branch Component: Runtime System| Version: 6.4.1 Severity: normal| Resolution: Keywords:| Difficulty: Moderate (1 day) Testcase: N/A | Os: Linux Architecture: Unknown/Multiple | -+-- Changes (by guest): * cc: gwe...@gmail.com (added) Comment: I second Bulat's comment. This is *very* important for long-running servers. Consider the Gitit wiki server. There are a few pages like 'Recent changes' which get the entire revision history; each time this happens, the memory usage goes up a bit - even though even last bit of the history will get discarded once the page has been constructed and sent off to the client. (We've checked for memory leaks.) This is especially problematic when Gitit is being used, surprisingly enough, as a web host, since typically this is done on virtualized slices or is otherwise resource-constrained. It looks bad for wiki.darcs.net that an idling gitit takes 31% of RAM. -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/698#comment:9 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #698: GHC's internal memory allocator never releases memory back to the OS
#698: GHC's internal memory allocator never releases memory back to the OS -+-- Reporter: guest |Owner: igloo Type: bug | Status: new Priority: low |Milestone: 6.12 branch Component: Runtime System| Version: 6.4.1 Severity: normal| Resolution: Keywords:| Difficulty: Moderate (1 day) Testcase: N/A | Os: Linux Architecture: Unknown/Multiple | -+-- Changes (by igloo): * owner: = igloo * milestone: 6.10 branch = 6.12 branch -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/698#comment:7 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #698: GHC's internal memory allocator never releases memory back to the OS
#698: GHC's internal memory allocator never releases memory back to the OS +--- Reporter: guest | Owner: Type: bug | Status: new Priority: low | Milestone: 6.10 branch Component: Runtime System |Version: 6.4.1 Severity: normal | Resolution: Keywords: | Difficulty: Moderate (1 day) Testcase: N/A | Architecture: Multiple Os: Linux | +--- Changes (by simonmar): * milestone: 6.8 branch = 6.10 branch -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/698#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #698: GHC's internal memory allocator never releases memory back to the OS
#698: GHC's internal memory allocator never releases memory back to the OS ---+ Reporter: guest |Owner: Type: bug | Status: new Priority: low |Milestone: 6.8 Component: Runtime System | Version: 6.4.1 Severity: normal | Resolution: Keywords: | Difficulty: Moderate (1 day) Os: Linux | Testcase: N/A Architecture: Multiple| ---+ Changes (by guest): * cc: = [EMAIL PROTECTED] Comment: it becomes important when we are short on physical memory - in this case pages with garbage are written to swapfile :( it also important for long-running programs -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/698 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #698: GHC's internal memory allocator never releases memory back to the OS
#698: GHC's internal memory allocator never releases memory back to the OS +--- Reporter: guest | Owner: Type: bug | Status: new Priority: low | Milestone: 6.8 Component: Runtime System |Version: 6.4.1 Severity: normal | Resolution: Keywords: | Difficulty: Moderate (1 day) Testcase: N/A | Architecture: Multiple Os: Linux | +--- Changes (by igloo): * milestone: = 6.8 * testcase: = N/A -- Ticket URL: http://hackage.haskell.org/trac/ghc/ticket/698 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #698: GHC's internal memory allocator never releases memory back to the OS
#698: GHC's internal memory allocator never releases memory back to the OS -+-- Reporter: guest | Owner: Type: bug | Status: new Priority: low | Milestone: Component: Runtime System |Version: 6.4.1 Severity: normal | Resolution: Keywords: | Os: Linux Difficulty: Unknown | Architecture: x86 -+-- Changes (by simonmar): * component: Compiler (FFI) = Runtime System * priority: normal = low * summary: allocaBytes does not actually free the memory after the computation = GHC's internal memory allocator never releases memory back to the OS Comment: I've changed $(subject), this bug is actually just an instance of the more general problem that GHC never releases any memory back to the OS. The memory allocated by allocaBytes has actually been freed, but it is still held by GHC's memory allocator, and not the OS. Normally this isn't a big problem, because the memory will be re-used by GHC. However, in cases where you have a large spike in memory use or want to temporarily allocate a very large object, it becomes noticeable. -- Ticket URL: http://cvs.haskell.org/trac/ghc/ticket/698 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
Re: [GHC] #698: GHC's internal memory allocator never releases memory back to the OS
#698: GHC's internal memory allocator never releases memory back to the OS ---+ Reporter: guest | Owner: Type: bug | Status: new Priority: low | Milestone: Component: Runtime System|Version: 6.4.1 Severity: normal| Resolution: Keywords:| Os: Linux Difficulty: Moderate (1 day) | Architecture: Multiple ---+ Changes (by simonmar): * architecture: x86 = Multiple * difficulty: Unknown = Moderate (1 day) -- Ticket URL: http://cvs.haskell.org/trac/ghc/ticket/698 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler___ Glasgow-haskell-bugs mailing list Glasgow-haskell-bugs@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
