On 06/12/2011, at 6:58 PM, Kris De Volder wrote: >> So what happens to this large chunk of memory we've claimed from the >> OS but don't go near again? My point was that the OS will probably >> page it out because we don't touch it. This is assuming quite a bit >> though. >> >> At no point did I say memory was free. My contention was that, given >> the memory profile of the daemon, we should be generous instead of >> conservative as being able to do more builds without having to >> change the defaults is more appealing than being memory conservative >> because of paging. > > Sorry I didn't want to offend. I just wanted to comment because my experience > with how it currently works is not great. Gradle daemons put serious pressure > on my memory and were literally making my 4Gb machine *unusable*, unless I > went out of my way to drop to OS and "kill -9" some gradle daemons every so > often. I actually have upgraded my machine from 4Gb to 8Gb because of Gradle. > > From my limited experience, counting on having the OS page out the unused > heap isn't really an option. It just doesn't work well, and with about 20% of > memory being 'swap' the UI becomes completely unresponsive. > > I can see the trade-off here is a tricky one. I alos agree with you, to err a > bit on the side of generousness, but *only* if the tooling API can do a very > good job of managing the daemons it spawns.
What kind of management does it need to do? i.e. what would very good management look like? > Another thing to consider: as you say, indeed the JVM never gives memory back > to the OS. The only way to really give it back is to terminate the JVM (i.e. > daemon). So I'd suggest not having the daemon stick around for too long, > unless it is really being used quite frequently… At the moment, the default idle time is 3 hours. This will be user configurable though. > My compromise would be a combination of two strategies: > - be generous with max memory limits (so things work without much manual > tweaking) > - be conservative and 'kill' unused daemons of fairly quickly (so unused > resources are returned to OS) This sounds right to me, now we just need to agree on what the idle timeout should be :) If we are too conservative we almost completely negate the user experience benefit of using the daemon. > I believe, it just isn't worth the price to keep a daemon parked in precious > memory for extended periods of time, if it is not being used. Rather than > counting on paging it out, its much better to shut it down. If it is paged > out, it won't give you great performance next time you try to use it anyway. It would be interesting to test if paging in is cheaper than bootstrapping Groovy or not. I'm not suggesting we do this. Shutting it down is costly though in terms of user experience so that's not without consequence. This is a fine balancing act. > Just my 2 cents. > > And sorry if I offended you. The reason why I feel strongly and reacted so > strongly is I have personally experienced that counting on the OS to page > out unused daemons doesn't seem to work very well. No problem, no offence taken. I just had to clarify what I was saying. Your input is very valuable and I think we are increasing the understanding here. Although, personally I don't think I have a good idea on these defaults yet. > Re: calling system.gc > > OK, maybe this is a good idea, or maybe not. > > I just warn, not to put in a call to force a gc unless you *really* are sure > it is good, and you know what you are doing, and what else may be going on on > the machine, outside the gradle daemon. > > For example, I'm pretty sure that *if* you are a in situation where heap is > partially paged out, this will cause massive paging, because the gc will > essentially touch all the paged out memory. > This will negatively impact every other process running on the machine. > > That may not even be so bad, if nothing else was running on the machine... > However, for a concrete counter-example (close to my heart :-). The IDE may > be waiting for the build's result. So a completed build may in fact be the > trigger for another process to start consuming the result of that build. In > such a scenario, the daemon may be idle, but the machine it is running on > certainly is not, and gradle daemon's System.gc will be competing for cycles > and memory space with other active processes. Given that we can't be _sure_ it's harmless, let's just leave it alone. -- Luke Daley Principal Engineer, Gradleware http://gradleware.com --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email
