Regarding memory usage, I suggest that an self-tuning algorithm is considered. During an "initial phase", it tries to find the OutOfMemory condition, by add/removing chunks (100MB?) of memory for the next run, starting at a very generous level (all system memory?). Once it has determined that it is "just above" a working situation it switches to "stable phase" and if OOM then increase by a smaller chunk (50MB maybe) to accommodating for increasing demands from project itself. This tuning could happen for PermGen space and heap space independently.
With this, the user get the best overall experience with occasional OOM and rebuild. Sounds like a bargain, and can't imaging the algorithm to be hard to implement. Cheers Niclas On Fri, Dec 9, 2011 at 9:44 AM, Adam Murdoch <[email protected]> wrote: > > On 07/12/2011, at 10:44 PM, Luke Daley wrote: > > > On 06/12/2011, at 6:58 PM, Kris De Volder wrote: > > So what happens to this large chunk of memory we've claimed from the > > OS but don't go near again? My point was that the OS will probably > > page it out because we don't touch it. This is assuming quite a bit > > though. > > > At no point did I say memory was free. My contention was that, given > > the memory profile of the daemon, we should be generous instead of > > conservative as being able to do more builds without having to > > change the defaults is more appealing than being memory conservative > > because of paging. > > > Sorry I didn't want to offend. I just wanted to comment because my > experience with how it currently works is not great. Gradle daemons put > serious pressure on my memory and were literally making my 4Gb machine > *unusable*, unless I went out of my way to drop to OS and "kill -9" some > gradle daemons every so often. I actually have upgraded my machine from 4Gb > to 8Gb because of Gradle. > > > From my limited experience, counting on having the OS page out the unused > heap isn't really an option. It just doesn't work well, and with about 20% > of memory being 'swap' the UI becomes completely unresponsive. > > > I can see the trade-off here is a tricky one. I alos agree with you, to err > a bit on the side of generousness, but *only* if the tooling API can do a > very good job of managing the daemons it spawns. > > > What kind of management does it need to do? i.e. what would very good > management look like? > > Another thing to consider: as you say, indeed the JVM never gives memory > back to the OS. The only way to really give it back is to terminate the JVM > (i.e. daemon). So I'd suggest not having the daemon stick around for too > long, unless it is really being used quite frequently… > > > At the moment, the default idle time is 3 hours. This will be user > configurable though. > > My compromise would be a combination of two strategies: > > - be generous with max memory limits (so things work without much manual > tweaking) > > - be conservative and 'kill' unused daemons of fairly quickly (so unused > resources are returned to OS) > > > This sounds right to me, now we just need to agree on what the idle timeout > should be :) > > If we are too conservative we almost completely negate the user experience > benefit of using the daemon. > > I believe, it just isn't worth the price to keep a daemon parked in precious > memory for extended periods of time, if it is not being used. Rather than > counting on paging it out, its much better to shut it down. If it is paged > out, it won't give you great performance next time you try to use it anyway. > > > It would be interesting to test if paging in is cheaper than bootstrapping > Groovy or not. I'm not suggesting we do this. > > Shutting it down is costly though in terms of user experience so that's not > without consequence. > > > Particularly as the daemon starts to do more stuff in the background: > * Up-to-date checks as files change > * Checking for and downloading changed dependencies. > * Pre-configuring the model ready for building. > * Caching the tooling API model. > * Pushing model deltas to the tooling API client. > * Assisting with tab completion on the command-line. > > > This is a fine balancing act. > > Just my 2 cents. > > > And sorry if I offended you. The reason why I feel strongly and reacted so > strongly is I have personally experienced that counting on the OS to page > out unused daemons doesn't seem to work very well. > > > No problem, no offence taken. I just had to clarify what I was saying. Your > input is very valuable and I think we are increasing the understanding here. > Although, personally I don't think I have a good idea on these defaults yet. > > Re: calling system.gc > > > OK, maybe this is a good idea, or maybe not. > > > I just warn, not to put in a call to force a gc unless you *really* are sure > it is good, and you know what you are doing, and what else may be going on > on the machine, outside the gradle daemon. > > > For example, I'm pretty sure that *if* you are a in situation where heap is > partially paged out, this will cause massive paging, because the gc will > essentially touch all the paged out memory. > > This will negatively impact every other process running on the machine. > > > That may not even be so bad, if nothing else was running on the machine... > However, for a concrete counter-example (close to my heart :-). The IDE may > be waiting for the build's result. So a completed build may in fact be the > trigger for another process to start consuming the result of that build. In > such a scenario, the daemon may be idle, but the machine it is running on > certainly is not, and gradle daemon's System.gc will be competing for cycles > and memory space with other active processes. > > > Given that we can't be _sure_ it's harmless, let's just leave it alone. > > -- > Luke Daley > Principal Engineer, Gradleware > http://gradleware.com > > > --------------------------------------------------------------------- > To unsubscribe from this list, please visit: > > http://xircles.codehaus.org/manage_email > > > > > -- > Adam Murdoch > Gradle Co-founder > http://www.gradle.org > VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting > http://www.gradleware.com > -- Niclas Hedhman, Software Developer http://www.qi4j.org - New Energy for Java I live here; http://tinyurl.com/3xugrbk I work here; http://tinyurl.com/6a2pl4j I relax here; http://tinyurl.com/2cgsug --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email
