On 07/12/2011, at 10:44 PM, Luke Daley wrote: > > On 06/12/2011, at 6:58 PM, Kris De Volder wrote: > >>> So what happens to this large chunk of memory we've claimed from the >>> OS but don't go near again? My point was that the OS will probably >>> page it out because we don't touch it. This is assuming quite a bit >>> though. >>> >>> At no point did I say memory was free. My contention was that, given >>> the memory profile of the daemon, we should be generous instead of >>> conservative as being able to do more builds without having to >>> change the defaults is more appealing than being memory conservative >>> because of paging. >> >> Sorry I didn't want to offend. I just wanted to comment because my >> experience with how it currently works is not great. Gradle daemons put >> serious pressure on my memory and were literally making my 4Gb machine >> *unusable*, unless I went out of my way to drop to OS and "kill -9" some >> gradle daemons every so often. I actually have upgraded my machine from 4Gb >> to 8Gb because of Gradle. >> >> From my limited experience, counting on having the OS page out the unused >> heap isn't really an option. It just doesn't work well, and with about 20% >> of memory being 'swap' the UI becomes completely unresponsive. >> >> I can see the trade-off here is a tricky one. I alos agree with you, to err >> a bit on the side of generousness, but *only* if the tooling API can do a >> very good job of managing the daemons it spawns. > > What kind of management does it need to do? i.e. what would very good > management look like? > >> Another thing to consider: as you say, indeed the JVM never gives memory >> back to the OS. The only way to really give it back is to terminate the JVM >> (i.e. daemon). So I'd suggest not having the daemon stick around for too >> long, unless it is really being used quite frequently… > > At the moment, the default idle time is 3 hours. This will be user > configurable though. > >> My compromise would be a combination of two strategies: >> - be generous with max memory limits (so things work without much manual >> tweaking) >> - be conservative and 'kill' unused daemons of fairly quickly (so unused >> resources are returned to OS) > > This sounds right to me, now we just need to agree on what the idle timeout > should be :) > > If we are too conservative we almost completely negate the user experience > benefit of using the daemon. > >> I believe, it just isn't worth the price to keep a daemon parked in precious >> memory for extended periods of time, if it is not being used. Rather than >> counting on paging it out, its much better to shut it down. If it is paged >> out, it won't give you great performance next time you try to use it anyway. > > It would be interesting to test if paging in is cheaper than bootstrapping > Groovy or not. I'm not suggesting we do this. > > Shutting it down is costly though in terms of user experience so that's not > without consequence.
Particularly as the daemon starts to do more stuff in the background: * Up-to-date checks as files change * Checking for and downloading changed dependencies. * Pre-configuring the model ready for building. * Caching the tooling API model. * Pushing model deltas to the tooling API client. * Assisting with tab completion on the command-line. > This is a fine balancing act. > >> Just my 2 cents. >> >> And sorry if I offended you. The reason why I feel strongly and reacted so >> strongly is I have personally experienced that counting on the OS to page >> out unused daemons doesn't seem to work very well. > > No problem, no offence taken. I just had to clarify what I was saying. Your > input is very valuable and I think we are increasing the understanding here. > Although, personally I don't think I have a good idea on these defaults yet. > >> Re: calling system.gc >> >> OK, maybe this is a good idea, or maybe not. >> >> I just warn, not to put in a call to force a gc unless you *really* are sure >> it is good, and you know what you are doing, and what else may be going on >> on the machine, outside the gradle daemon. >> >> For example, I'm pretty sure that *if* you are a in situation where heap is >> partially paged out, this will cause massive paging, because the gc will >> essentially touch all the paged out memory. >> This will negatively impact every other process running on the machine. >> >> That may not even be so bad, if nothing else was running on the machine... >> However, for a concrete counter-example (close to my heart :-). The IDE may >> be waiting for the build's result. So a completed build may in fact be the >> trigger for another process to start consuming the result of that build. In >> such a scenario, the daemon may be idle, but the machine it is running on >> certainly is not, and gradle daemon's System.gc will be competing for cycles >> and memory space with other active processes. > > Given that we can't be _sure_ it's harmless, let's just leave it alone. > > -- > Luke Daley > Principal Engineer, Gradleware > http://gradleware.com > > > --------------------------------------------------------------------- > To unsubscribe from this list, please visit: > > http://xircles.codehaus.org/manage_email > > -- Adam Murdoch Gradle Co-founder http://www.gradle.org VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting http://www.gradleware.com
